:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(tracing): add apm-service-health and reject malformed GET date ranges (#119975)
Daniel Visca86a32f059c2f
diff --git a/posthog/test/repo_invariants/signals_scout_tool_citations_baseline.txt b/posthog/test/repo_invariants/signals_scout_tool_citations_baseline.txtindex e4250b41769c..8488a6ce2b28 100644--- a/posthog/test/repo_invariants/signals_scout_tool_citations_baseline.txt+++ b/posthog/test/repo_invariants/signals_scout_tool_citations_baseline.txt@@ -1,6 +1,7 @@agent-feedbackalert-simulateapm-attribute-breakdown+apm-service-healthapm-spans-aggregateapm-spans-duration-histogramapm-spans-sparklinediff --git a/products/signals/skills/signals-scout-apm/SKILL.md b/products/signals/skills/signals-scout-apm/SKILL.mdindex 311576987dba..bb84908ccbfc 100644--- a/products/signals/skills/signals-scout-apm/SKILL.md+++ b/products/signals/skills/signals-scout-apm/SKILL.md@@ -7,8 +7,8 @@ description: >compatibility: >PostHog Signals agent (Claude sandbox). Read-only analytics + signal_scout_internal:write(scratchpad) + signal_scout_report:write (report channel), plus the apm-* tool family- (query-apm-spans, apm-trace-get, apm-spans-aggregate, apm-spans-tree, apm-spans-count,- apm-spans-sparkline, apm-spans-duration-histogram, apm-attribute-breakdown,+ (apm-service-health, query-apm-spans, apm-trace-get, apm-spans-aggregate, apm-spans-tree,+ apm-spans-count, apm-spans-sparkline, apm-spans-duration-histogram, apm-attribute-breakdown,apm-services-list, apm-attributes-list, apm-attribute-values-list) and the bundledexploring-apm-traces deep-dive skill.allowed_tools:@@ -60,30 +60,28 @@ Three cheap reads cold-start a run:- `scout-scratchpad-search` (`text=apm`) — durable steering from past APM runs. **Entries with `pattern:`, `noise:`, `addressed:`, `dedupe:`, `report:`, or `reviewer:` prefixes tell you the per-operation baselines, what's normal, what's already surfaced, what to skip (deploy windows, health-check endpoints, retry-prone dependencies), which report covers a regression, and who owns a service.**- `scout-runs-list` (last 7d) — what prior APM runs found and ruled out. Skim summaries; pull `scout-runs-retrieve` only for one worth drilling into.-- `apm-services-list` — the live service inventory. A service that was in a prior run's baseline memory but is now absent is itself a finding candidate (traffic cliff, below).+- `apm-service-health` — the live service inventory with a status per service (the discriminator engine, below). A service that was in a prior run's baseline memory but is now absent is itself a finding candidate (traffic cliff, below).- `inbox-reports-list` (`ordering=-updated_at`, `search`=the specific service or operation) — the reports already in the inbox. Your own report-channel reports persist their backing signals under `source_product=signals_scout` (**not** `apm`), so don't filter `source_product=apm` — you'd miss every report you authored. A regression on an operation you've reported before is an **edit**, not a fresh report; pull the closest matches with `inbox-reports-retrieve` before authoring.### The discriminator engine-One call gives you the seasonality-matched baseline for every operation:+One call scores every service against the same window one week ago:```json-apm-spans-aggregate+apm-service-health{"query": {"dateRange": { "date_from": "-1d" },- "compareFilter": { "compare": true, "compare_to": "-7d" }+ "compare": "previous_week"}}```-`results` is the last 24h, `compare` is the same 24h one week ago — both as one row per `(service_name, name)` with `count`, `error_count`, `p50_duration_nano`, `p95_duration_nano`. Join the two arrays on `(service_name, name)` and compute, per operation:+Each `services` row has `requests` and `baseline_requests` (the denominator guard), `error_rate_pct` and `error_rate_delta_pp`, `p95_ms` and `p95_change_pct`, and a `status` with a `reason`. The server computes the service percentiles, so do not rebuild them from operation rows. Rows come worst first: `silent`, `regressed`, `degraded`. The `next` calls open the operations behind each regressed or degraded service. Exclude dev and staging with `environment` when the project labels it.-- **error rate** `error_count / count`, now vs 7d-ago-- **p95 latency** `p95_duration_nano`, now vs 7d-ago-- **request volume** `count`, now vs 7d-ago (the denominator guard)+The server does not apply the volume guard. For each `regressed` row, check that `requests` stayed within ~2× of `baseline_requests` before you treat it as a regression.-A busy service returns hundreds of operations (the payload runs to 100KB+ and the harness persists it to a file) — **process it programmatically, don't eyeball it.** Sort operations by delta and keep only those where the rate moved but `count` stayed within ~2× (the guard that separates a real regression from a volume swing); a low-`count` operation has too small a sample for a stable percentile (see disqualifiers). Scope to a few services per run rather than pulling the whole project at once.+Then drill into the operations of the regressed services only: run the `apm-spans-aggregate` call in the response's `next` (it is already scoped to those services, with `compareFilter` on the same baseline). With `environment` set, `next` is empty, so build that call yourself. Do not pull the whole project per operation and join the arrays — a busy project returns 100KB+ that you must then process by hand.### Profile shape@@ -94,7 +92,7 @@ A busy service returns hundreds of operations (the payload runs to 100KB+ and th| `p95` up materially, `count` ~flat | latency regression — investigate || `p95` up **and** `count` up sharply | saturation under load — investigate, lower confidence || new `(service, name)` erroring, no 7d-ago row | new code path / recent deploy — investigate |-| service in baseline memory, now ~0 spans | traffic cliff (instrumentation break or outage) — investigate |+| service `status: silent` | traffic cliff (instrumentation break or outage) — investigate |Always score the **latest complete** bucket/window — a partial current hour always reads as a drop in volume and a dip in p95.@@ -104,7 +102,7 @@ Patterns to watch — starting points, not a checklist.#### Error-rate regression-From the discriminator engine, find operations where error rate stepped up materially while `count` held roughly steady. Confirm _when_ it started: `apm-spans-sparkline` with your service/operation filters for total counts, then the same call with `statusCodes: [2]` for error counts — error rate per bucket = errors / total; the bucket where the ratio jumps is the onset. Pull a representative failing trace: `query-apm-spans` with a `status_code = 2` filter and `orderBy: "duration"`, grab a `trace_id`, then `apm-trace-get` and read `exception.type` / `exception.message` straight off the error span's `attributes` map. Walk `parent_span_id` up to see the request path that led there. **`query-apm-spans` defaults to root spans only** (`rootSpans: true`), so when the regressed operation is a child span (a DB or `Client` call), set `flatSpans: true` (and `rootSpans: false`) or the `status_code = 2` + operation filter matches nothing — the aggregate flags the regression but you can never pull a sample to confirm it.+From the discriminator engine, take the `regressed` services whose `reason` names the error rate, and find the operations in them where the error rate stepped up while `count` held roughly steady. Confirm _when_ it started: `apm-spans-sparkline` with your service/operation filters for total counts, then the same call with `statusCodes: [2]` for error counts — error rate per bucket = errors / total; the bucket where the ratio jumps is the onset. Pull a representative failing trace: `query-apm-spans` with a `status_code = 2` filter and `orderBy: "duration"`, grab a `trace_id`, then `apm-trace-get` and read `exception.type` / `exception.message` straight off the error span's `attributes` map. Walk `parent_span_id` up to see the request path that led there. **`query-apm-spans` defaults to root spans only** (`rootSpans: true`), so when the regressed operation is a child span (a DB or `Client` call), set `flatSpans: true` (and `rootSpans: false`) or the `status_code = 2` + operation filter matches nothing — the aggregate flags the regression but you can never pull a sample to confirm it.#### Latency p95 regression@@ -118,7 +116,7 @@ An operation (or a downstream `Client`-kind span calling another service) newly#### Service traffic cliff-Compare `apm-services-list` and per-service `apm-spans-sparkline` against baseline memory: a service that emitted a steady span volume and dropped to ~0 is an instrumentation break or an outage (the trace-side analog of a capture cliff — spans are not retroactive). Guard against reading a partial current bucket as a cliff: confirm the drop spans ≥2 complete buckets.+A `silent` row in `apm-service-health` is a service that had a steady span volume and stopped: an instrumentation break or an outage (the trace-side analog of a capture cliff — spans are not retroactive). Its `reason` names the last span. The server calls a service silent only when its spans stopped for at least 10 minutes (two 5-minute buckets) where its own rate predicted more, so one quiet partial bucket does not count. Date the drop with `apm-spans-sparkline` on that service. Also check baseline memory for services absent from both windows.### Save memory as you go@@ -160,7 +158,7 @@ When in doubt, write memory instead of filing a report.## MCP tools-Direct (read-only): `apm-services-list`, `apm-spans-aggregate`, `apm-spans-sparkline`, `apm-spans-tree`, `apm-spans-duration-histogram`, `apm-attribute-breakdown`, `apm-attributes-list`, `apm-attribute-values-list`, `apm-spans-count`, `query-apm-spans`, `apm-trace-get`.+Direct (read-only): `apm-service-health`, `apm-services-list`, `apm-spans-aggregate`, `apm-spans-sparkline`, `apm-spans-tree`, `apm-spans-duration-histogram`, `apm-attribute-breakdown`, `apm-attributes-list`, `apm-attribute-values-list`, `apm-spans-count`, `query-apm-spans`, `apm-trace-get`.Inbox & reviewer routing (mechanics in `authoring-scouts` → `references/report-contract.md`):diff --git a/products/tracing/backend/facade/api.py b/products/tracing/backend/facade/api.pyindex 37ab68d2f872..ccadbc234396 100644--- a/products/tracing/backend/facade/api.py+++ b/products/tracing/backend/facade/api.py@@ -17,7 +17,7 @@ Do NOT:"""from datetime import datetime-from typing import TYPE_CHECKING+from typing import TYPE_CHECKING, Anyfrom posthog.schema import (CachedTraceSpansAttributeBreakdownQueryResponse,@@ -58,8 +58,14 @@ from products.tracing.backend.latency_heatmap_query_runner import (run_latency_heatmap_query as _run_latency_heatmap_query,)from products.tracing.backend.self_time import annotate_self_time as _annotate_self_time+from products.tracing.backend.service_health_query_runner import (+ run_service_health_query as _run_service_health_query,+ service_health_byte_cap_message as _service_health_byte_cap_message,+)from products.tracing.backend.symbol_stats_query_runner import run_symbol_stats_query as _run_symbol_stats_query+from .contracts import ServiceHealthParams+if TYPE_CHECKING:from posthog.models import Team, User@@ -258,3 +264,11 @@ def fetch_trace_ai_events(*, team: "Team", user: "User | None", trace_id: str) -the returned columns."""return _fetch_trace_ai_events(team=team, user=user, trace_id=trace_id)+++def run_service_health_query(*, team: "Team", params: ServiceHealthParams) -> dict[str, Any]:+ return _run_service_health_query(team=team, params=params)+++def service_health_byte_cap_message(*, team: "Team", params: ServiceHealthParams) -> str:+ return _service_health_byte_cap_message(team=team, params=params)diff --git a/products/tracing/backend/facade/contracts.py b/products/tracing/backend/facade/contracts.pyindex 7a482010b208..3dcfedc0dc09 100644--- a/products/tracing/backend/facade/contracts.py+++ b/products/tracing/backend/facade/contracts.py@@ -11,3 +11,21 @@ Characteristics:Do NOT depend on Django models, DRF serializers, or request objects."""++from posthog.dataclasses import frozen++from products.tracing.backend.facade import enums+++@frozen+class ServiceHealthParams:+ date_from: str = "-1h"+ date_to: str | None = None+ compare: enums.ServiceHealthCompare = enums.ServiceHealthCompare.PREVIOUS_PERIOD+ compare_date_from: str | None = None+ compare_date_to: str | None = None+ environment: str | None = None+ service_names: tuple[str, ...] = ()+ span_scope: enums.ServiceHealthSpanScope = enums.ServiceHealthSpanScope.ENTRY+ order_by: enums.ServiceHealthOrderBy = enums.ServiceHealthOrderBy.STATUS+ limit: int = 25diff --git a/products/tracing/backend/facade/enums.py b/products/tracing/backend/facade/enums.pyindex dc25d9a5eebe..4417d30849f0 100644--- a/products/tracing/backend/facade/enums.py+++ b/products/tracing/backend/facade/enums.py@@ -11,3 +11,43 @@ products need to interpret contract objects also belong here.Internal-only constants (DB magic values, feature flags, etc.)should stay in the implementation (logic.py, models.py)."""++from posthog.enums import LabeledStrEnum+++class ServiceHealthCompare(LabeledStrEnum):+ NONE = "none"+ PREVIOUS_PERIOD = "previous_period"+ PREVIOUS_DAY = "previous_day"+ PREVIOUS_WEEK = "previous_week"+ CUSTOM = "custom"+++class ServiceHealthBaseline(LabeledStrEnum):+ NONE = "none"+ PREVIOUS_PERIOD = "previous_period"+ PREVIOUS_DAY = "previous_day"+ PREVIOUS_WEEK = "previous_week"+++class ServiceHealthSpanScope(LabeledStrEnum):+ ENTRY = "entry"+ ALL = "all"+++class ServiceHealthOrderBy(LabeledStrEnum):+ STATUS = "status"+ ERROR_RATE = "error_rate"+ P95 = "p95"+ THROUGHPUT = "throughput"+ ERROR_COUNT = "error_count"+++class ServiceHealthStatus(LabeledStrEnum):+ SILENT = "silent"+ REGRESSED = "regressed"+ DEGRADED = "degraded"+ NEW = "new"+ RECOVERED = "recovered"+ IMPROVED = "improved"+ OK = "ok"diff --git a/products/tracing/backend/presentation/date_window.py b/products/tracing/backend/presentation/date_window.pyindex 4d731af56c63..637e341611ba 100644--- a/products/tracing/backend/presentation/date_window.py+++ b/products/tracing/backend/presentation/date_window.py@@ -17,10 +17,13 @@ previously turned a typo'd window into stale data with no error."""import re+from zoneinfo import ZoneInfofrom dateutil import parser as dateutil_parserfrom rest_framework import serializers+from posthog.utils import relative_date_parse+# Friendly tracing window: an integer with a single short unit, e.g. "-30m", "90s", "-2w".# Case-insensitive; the leading minus is optional because windows are always in the past._TRACING_WINDOW_RE = re.compile(r"^-?(?P<number>\d+)(?P<unit>[smhdw])$", re.IGNORECASE)@@ -37,6 +40,14 @@ _UNIT_TO_RELATIVE_TOKEN = {"s": "s", "m": "M", "h": "h", "d": "d", "w": "w"}_GLOBAL_RELATIVE_RE = re.compile(r"-?(?:\d+[hdwmqysHDWMQY](?:Start|End)?|[hdwmqysHDWMQY](?:Start|End))")+def _in_range(window: str) -> str:+ try:+ relative_date_parse(window, ZoneInfo("UTC"))+ except (OverflowError, ValueError):+ raise serializers.ValidationError(f"Date range value {window!r} is out of range.")+ return window++def normalize_tracing_window(value: str | None) -> str | None:"""Normalize a single ``date_from``/``date_to`` value for the tracing API.@@ -46,6 +57,8 @@ def normalize_tracing_window(value: str | None) -> str | None:"""if value is None:return None+ if not isinstance(value, str):+ raise serializers.ValidationError(f"Invalid date range value {value!r}: dates must be strings.")stripped = value.strip()if not stripped:return None@@ -53,19 +66,22 @@ def normalize_tracing_window(value: str | None) -> str | None:match = _TRACING_WINDOW_RE.match(stripped)if match:token = _UNIT_TO_RELATIVE_TOKEN[match.group("unit").lower()]- return f"-{match.group('number')}{token}"+ return _in_range(f"-{match.group('number')}{token}")# Absolute ISO 8601 timestamps pass through untouched.try:- dateutil_parser.isoparse(stripped)- return stripped+ parsed = dateutil_parser.isoparse(stripped)except ValueError:- pass+ parsed = None+ if parsed is not None:+ if not 1970 <= parsed.year <= 9998:+ raise serializers.ValidationError(f"Date range value {value!r} is out of range.")+ return stripped# Preserve the broader global relative grammar (e.g. "-1dStart", "wStart", "-3q", "-1y")# so anything that already worked keeps working.if _GLOBAL_RELATIVE_RE.fullmatch(stripped):- return stripped+ return _in_range(stripped)raise serializers.ValidationError(f"Invalid date range value {value!r}. Use a relative window like '-30m' (minutes), "diff --git a/products/tracing/backend/presentation/service_health_serializers.py b/products/tracing/backend/presentation/service_health_serializers.pynew file mode 100644index 000000000000..423a93478f8c--- /dev/null+++ b/products/tracing/backend/presentation/service_health_serializers.py@@ -0,0 +1,170 @@+import datetime as dt+from zoneinfo import ZoneInfo++from drf_spectacular.utils import extend_schema_field+from rest_framework import serializers++from posthog.utils import relative_date_parse++from ..facade.contracts import ServiceHealthParams+from ..facade.enums import (+ ServiceHealthBaseline,+ ServiceHealthCompare,+ ServiceHealthOrderBy,+ ServiceHealthSpanScope,+ ServiceHealthStatus,+)+from .date_window import normalize_tracing_window++UTC = ZoneInfo("UTC")+++class _ServiceHealthDateRangeSerializer(serializers.Serializer):+ date_from = serializers.CharField(+ required=False, allow_null=True, help_text="Relative (-30m, -6h, -7d) or ISO 8601."+ )+ date_to = serializers.CharField(required=False, allow_null=True, help_text='Same format. Omit or null for "now".')+++class _ServiceHealthQuerySerializer(serializers.Serializer):+ dateRange = _ServiceHealthDateRangeSerializer(+ required=False, help_text="The window to score. Defaults to the last hour."+ )+ compare = serializers.ChoiceField(+ choices=ServiceHealthBaseline.choices,+ default=ServiceHealthBaseline.PREVIOUS_PERIOD,+ help_text=(+ "The baseline. previous_period (default) is the same length just before the window. previous_day and "+ "previous_week shift the window back 1 or 7 days. none skips it. baselineDateRange overrides it."+ ),+ )+ environment = serializers.CharField(+ required=False,+ allow_null=True,+ allow_blank=True,+ help_text="Only this environment, matched without case on deployment.environment.name, then deployment.environment, then env.",+ )+ serviceNames = serializers.ListField(+ child=serializers.CharField(), required=False, default=list, max_length=100, help_text="Only these services."+ )+ spanScope = serializers.ChoiceField(+ choices=ServiceHealthSpanScope.choices,+ default=ServiceHealthSpanScope.ENTRY,+ help_text="entry (default) counts root, server and consumer spans, one per request. all counts every span.",+ )+ orderBy = serializers.ChoiceField(+ choices=ServiceHealthOrderBy.choices,+ default=ServiceHealthOrderBy.STATUS,+ help_text="status (default) puts silent, regressed and degraded first. The others sort high to low.",+ )+ limit = serializers.IntegerField(default=25, min_value=1, max_value=200, help_text="Max service rows, 1-200.")+++class ServiceHealthRequestSerializer(serializers.Serializer):+ query = _ServiceHealthQuerySerializer(help_text="The service health query. Every field is optional.")+ baselineDateRange = _ServiceHealthDateRangeSerializer(+ required=False, help_text="An explicit baseline window. Overrides query.compare."+ )++ def validate(self, attrs: dict) -> dict:+ for name, window in (+ ("dateRange", attrs["query"].get("dateRange")),+ ("baselineDateRange", attrs.get("baselineDateRange")),+ ):+ if window is None:+ continue+ start = normalize_tracing_window(window.get("date_from")) or "-1h"+ end = normalize_tracing_window(window.get("date_to"))+ if relative_date_parse(start, UTC) >= (relative_date_parse(end, UTC) if end else dt.datetime.now(UTC)):+ raise serializers.ValidationError({name: "date_from must be before date_to."})+ return attrs+++def service_health_params(validated_data: dict) -> ServiceHealthParams:+ data = validated_data["query"]+ date_range = data.get("dateRange") or {}+ baseline = validated_data.get("baselineDateRange")+ compare = ServiceHealthCompare.CUSTOM if baseline else ServiceHealthCompare(data["compare"])+ compare_from, compare_to = (baseline.get("date_from"), baseline.get("date_to")) if baseline else (None, None)+ return ServiceHealthParams(+ date_from=normalize_tracing_window(date_range.get("date_from")) or "-1h",+ date_to=normalize_tracing_window(date_range.get("date_to")),+ compare=compare,+ compare_date_from=normalize_tracing_window(compare_from),+ compare_date_to=normalize_tracing_window(compare_to),+ environment=(data.get("environment") or "").strip() or None,+ service_names=tuple(data["serviceNames"]),+ span_scope=ServiceHealthSpanScope(data["spanScope"]),+ order_by=ServiceHealthOrderBy(data["orderBy"]),+ limit=data["limit"],+ )+++class _ServiceHealthTimeRangeSerializer(serializers.Serializer):+ date_from = serializers.CharField(help_text="Start, ISO 8601 UTC, inclusive.")+ date_to = serializers.CharField(help_text="End, ISO 8601 UTC, exclusive.")+++class _ServiceHealthWindowSerializer(serializers.Serializer):+ current = _ServiceHealthTimeRangeSerializer(+ help_text="The scored window. Reuse it as dateRange in follow-up calls."+ )+ baseline = _ServiceHealthTimeRangeSerializer(+ allow_null=True, help_text="The baseline window. Null when compare is none."+ )+ timezone = serializers.CharField(help_text="Always UTC.")+++class _ServiceHealthSummarySerializer(serializers.Serializer):+ services = serializers.IntegerField(help_text="Services scored, before the limit.")+ requests = serializers.IntegerField(help_text="Requests over all scored services.")+ ok = serializers.IntegerField(help_text="Services per status.")+ degraded = serializers.IntegerField(help_text="Services per status.")+ regressed = serializers.IntegerField(help_text="Services per status.")+ silent = serializers.IntegerField(help_text="Services per status.")+ new = serializers.IntegerField(help_text="Services per status.")+ recovered = serializers.IntegerField(help_text="Services per status.")+ improved = serializers.IntegerField(help_text="Services per status.")+++class _ServiceHealthRowSerializer(serializers.Serializer):+ service = serializers.CharField(help_text="Service name.")+ status = serializers.ChoiceField(+ choices=ServiceHealthStatus.choices,+ help_text=(+ "silent: spans stopped. regressed: error rate or p95 worse than the baseline. degraded: error rate 5% or "+ "more. new: no spans in the baseline. recovered, improved: the reverse of regressed."+ ),+ )+ reason = serializers.CharField(allow_blank=True, help_text="Why the service has this status. Empty for ok.")+ requests = serializers.IntegerField(help_text="Spans in the window, in the chosen spanScope.")+ throughput_per_min = serializers.FloatField(help_text="Requests per minute.")+ error_count = serializers.IntegerField(help_text="Requests with OTel status Error.")+ error_rate_pct = serializers.FloatField(allow_null=True, help_text="Error share of requests, percent.")+ error_rate_delta_pp = serializers.FloatField(allow_null=True, help_text="Change from the baseline, points.")+ p50_ms = serializers.FloatField(allow_null=True, help_text="Median duration, ms.")+ p95_ms = serializers.FloatField(allow_null=True, help_text="95th percentile duration, ms.")+ p95_change_pct = serializers.FloatField(allow_null=True, help_text="p95 change from the baseline, percent.")+ p99_ms = serializers.FloatField(allow_null=True, help_text="99th percentile duration, ms.")+ baseline_requests = serializers.IntegerField(allow_null=True, help_text="Baseline requests. Null without one.")+ last_seen = serializers.CharField(allow_null=True, help_text="Last span, ISO 8601 UTC.")+++@extend_schema_field({"type": "object", "additionalProperties": True})+class _NextQueryField(serializers.JSONField):+ pass+++class _ServiceHealthNextCallSerializer(serializers.Serializer):+ tool = serializers.CharField(help_text="The tool to call next.")+ why = serializers.CharField(help_text="What the call answers.")+ query = _NextQueryField(help_text="The prefilled query argument.")+++class ServiceHealthResponseSerializer(serializers.Serializer):+ window = _ServiceHealthWindowSerializer(help_text="The absolute UTC windows the server used.")+ summary = _ServiceHealthSummarySerializer(help_text="Counts over all scored services.")+ services = _ServiceHealthRowSerializer(many=True, help_text="One row per service, in orderBy order.")+ next = _ServiceHealthNextCallSerializer(many=True, help_text="Prefilled follow-up calls.")+ truncated = serializers.BooleanField(help_text="True when more services matched than limit.")+ notes = serializers.ListField(child=serializers.CharField(), help_text="Caveats about this result.")diff --git a/products/tracing/backend/presentation/views.py b/products/tracing/backend/presentation/views.pyindex 24dc0e37a76c..8f8deee53da6 100644--- a/products/tracing/backend/presentation/views.py+++ b/products/tracing/backend/presentation/views.py@@ -64,7 +64,9 @@ from ..facade.api import (run_duration_histogram_query,run_impact_query,run_latency_heatmap_query,+ run_service_health_query,run_symbol_stats_query,+ service_health_byte_cap_message,)from ..has_spans_query_runner import team_has_spansfrom ..logic import (@@ -79,6 +81,11 @@ from ..logic import ()from ..sparkline_query_runner import TraceSpansSparklineQueryRunnerfrom .date_window import normalize_tracing_date_range+from .service_health_serializers import (+ ServiceHealthRequestSerializer,+ ServiceHealthResponseSerializer,+ service_health_params,+)# Matches FEATURE_FLAGS.TRACING_AI_EVENTS in the frontend.TRACING_AI_EVENTS_FEATURE_FLAG = "tracing-ai-events"@@ -384,11 +391,34 @@ class _TracingTraceRequestSerializer(serializers.Serializer):)-class _TracingServiceNamesQuerySerializer(serializers.Serializer):- search = serializers.CharField(required=False, help_text="Search filter for service names.")+class _TracingDateQueryParamsSerializer(serializers.Serializer):dateRange = serializers.CharField(required=False,- help_text='JSON-encoded date range, e.g. \'{"date_from": "-1h"}\'.',+ help_text='JSON-encoded date range object, e.g. {"date_from": "-1h"}. Malformed JSON returns 400. '+ "Or pass date_from and date_to.",+ )+ date_from = serializers.CharField(required=False, help_text="Range start: -1h, -7d or ISO 8601. Defaults to -1h.")+ date_to = serializers.CharField(required=False, help_text='Range end, same format. Omit for "now".')+++class _TracingServiceNamesQuerySerializer(_TracingDateQueryParamsSerializer):+ search = serializers.CharField(required=False, help_text="Search filter for service names.")+++def _date_range_from_query_params(request: Request) -> dict:+ raw = request.GET.get("dateRange")+ if raw:+ try:+ parsed = json.loads(raw)+ except json.JSONDecodeError:+ parsed = None+ if not isinstance(parsed, dict):+ example = json.dumps({"date_from": "-1h"})+ message = f"dateRange must be a JSON object such as {example}, or pass date_from and date_to. Got {raw!r}."+ raise serializers.ValidationError({"dateRange": message})+ return normalize_tracing_date_range(parsed)+ return normalize_tracing_date_range(+ {"date_from": request.GET.get("date_from"), "date_to": request.GET.get("date_to")})@@ -469,7 +499,7 @@ class _TracingErrorCountsResponseSerializer(serializers.Serializer):)-class _TracingAttributesQuerySerializer(serializers.Serializer):+class _TracingAttributesQuerySerializer(_TracingDateQueryParamsSerializer):search = serializers.CharField(required=False, help_text="Search filter for attribute names.")search_values = serializers.BooleanField(required=False,@@ -514,7 +544,7 @@ class SpanPropertyType(models.TextChoices):SPAN_RESOURCE_ATTRIBUTE = "span_resource_attribute", "span_resource_attribute"-class _TracingValuesQuerySerializer(serializers.Serializer):+class _TracingValuesQuerySerializer(_TracingDateQueryParamsSerializer):key = serializers.CharField(help_text="The attribute key to get values for.")attribute_type = serializers.ChoiceField(choices=SpanPropertyType.choices,@@ -1245,11 +1275,7 @@ class SpansViewSet(TeamAndOrgViewSetMixin, PydanticModelMixin, viewsets.ViewSet)def service_names(self, request: Request, *args, **kwargs) -> Response:tag_queries(product=ProductKey.TRACING, feature=Feature.QUERY)search = request.GET.get("search", "")- try:- raw_date_range = json.loads(request.GET.get("dateRange", '{"date_from": "-1h"}'))- except json.JSONDecodeError:- raw_date_range = {"date_from": "-1h"}- date_range = self.get_model(normalize_tracing_date_range(raw_date_range), DateRange)+ date_range = self.get_model(_date_range_from_query_params(request), DateRange)results = run_service_names_query(team=self.team, date_range=date_range, search=search)return Response({"results": results}, status=status.HTTP_200_OK)@@ -1703,6 +1729,36 @@ class SpansViewSet(TeamAndOrgViewSetMixin, PydanticModelMixin, viewsets.ViewSet)status=status.HTTP_200_OK,)+ @validated_request(+ ServiceHealthRequestSerializer,+ responses={200: OpenApiResponse(response=ServiceHealthResponseSerializer)},+ )+ @action(detail=False, methods=["POST"], required_scopes=["tracing:read"])+ def service_health(self, request: ValidatedRequest, *args, **kwargs) -> Response:+ tag_queries(product=ProductKey.TRACING, feature=Feature.QUERY)+ params = service_health_params(request.validated_data)+ try:+ result = run_service_health_query(team=self.team, params=params)+ except CHQueryErrorTooManyBytes:+ return Response(+ {"detail": service_health_byte_cap_message(team=self.team, params=params)},+ status=status.HTTP_400_BAD_REQUEST,+ )++ self._report_usage(+ request,+ "tracing service health queried",+ {+ "compare": params.compare.value,+ "span_scope": params.span_scope.value,+ "has_environment": bool(params.environment),+ "service_names_count": len(params.service_names),+ "services_count": result["summary"]["services"],+ "truncated": result["truncated"],+ },+ )+ return Response(ServiceHealthResponseSerializer(instance=result).data, status=status.HTTP_200_OK)+@extend_schema(request=_TracingTreeRequestSerializer,responses={200: _TracingTreeResponseSerializer},@@ -1973,11 +2029,7 @@ class SpansViewSet(TeamAndOrgViewSetMixin, PydanticModelMixin, viewsets.ViewSet)limit = int(request.GET.get("limit", "100"))offset = int(request.GET.get("offset", "0"))- try:- raw_date_range = json.loads(request.GET.get("dateRange", "{}"))- except json.JSONDecodeError:- raw_date_range = {}- date_range = self.get_model(normalize_tracing_date_range(raw_date_range), DateRange)+ date_range = self.get_model(_date_range_from_query_params(request), DateRange)attribute_type = request.GET.get("attribute_type", "span_attribute")if attribute_type not in ("span_attribute", "span_resource_attribute"):@@ -2007,11 +2059,7 @@ class SpansViewSet(TeamAndOrgViewSetMixin, PydanticModelMixin, viewsets.ViewSet)limit = int(request.GET.get("limit", "100"))offset = int(request.GET.get("offset", "0"))- try:- raw_date_range = json.loads(request.GET.get("dateRange", "{}"))- except json.JSONDecodeError:- raw_date_range = {}- date_range = self.get_model(normalize_tracing_date_range(raw_date_range), DateRange)+ date_range = self.get_model(_date_range_from_query_params(request), DateRange)attribute_type = request.GET.get("attribute_type", "span_attribute")if attribute_type not in ("span", "span_attribute", "span_resource_attribute"):diff --git a/products/tracing/backend/service_health_query_runner.py b/products/tracing/backend/service_health_query_runner.pynew file mode 100644index 000000000000..9e8980a81de4--- /dev/null+++ b/products/tracing/backend/service_health_query_runner.py@@ -0,0 +1,375 @@+import datetime as dt+from typing import TYPE_CHECKING, Any+from zoneinfo import ZoneInfo++from posthog.schema import DateRange, IntervalType, PropertyGroupsMode++from posthog.hogql import ast+from posthog.hogql.constants import LimitContext+from posthog.hogql.modifiers import create_default_modifiers_for_team+from posthog.hogql.parser import parse_expr, parse_select+from posthog.hogql.query import execute_hogql_query++from posthog.clickhouse.client.connection import Workload+from posthog.dataclasses import frozen+from posthog.hogql_queries.utils.caller_context import map_in_caller_context+from posthog.hogql_queries.utils.query_date_range import QueryDateRange++from products.tracing.backend.facade.contracts import ServiceHealthParams+from products.tracing.backend.facade.enums import (+ ServiceHealthCompare as Compare,+ ServiceHealthOrderBy as OrderBy,+ ServiceHealthSpanScope,+ ServiceHealthStatus as Status,+)+from products.tracing.backend.logic import TIME_BUCKET_DATE_RANGE_WHERE, fail_fast_scalar_settings++if TYPE_CHECKING:+ from posthog.models import Team++UTC = ZoneInfo("UTC")++SILENT_MIN_BASELINE_REQUESTS = 20+SILENT_GAP = dt.timedelta(minutes=10)+SILENT_MIN_EXPECTED_IN_GAP = 10+COMPARE_MIN_REQUESTS = 100+ERROR_RATE_DELTA_PP = 1.0+ERROR_RATE_RATIO = 2.0+P95_RISE_PCT = 50.0+DEGRADED_ERROR_RATE_PCT = 5.0+DEGRADED_MIN_REQUESTS = 20+MAX_REASON_LENGTH = 80+SERVICE_SCAN_LIMIT = 1000+NEXT_SERVICES_LIMIT = 5++_ENTRY_WHERE = "(is_root_span = 1 OR kind IN (2, 5))"+_ENVIRONMENT = (+ "coalesce(nullIf(resource_attributes['deployment.environment.name'], ''), "+ "nullIf(resource_attributes['deployment.environment'], ''), resource_attributes['env'], '')"+)+_ENVIRONMENT_WHERE = "lower(" + _ENVIRONMENT + ") = {env}"+_STATUS_ORDER = list(Status)+_UNHEALTHY = (Status.REGRESSED, Status.DEGRADED)+_SHIFT = {Compare.PREVIOUS_DAY: dt.timedelta(days=1), Compare.PREVIOUS_WEEK: dt.timedelta(days=7)}+_AGGREGATE_COMPARE_FILTER: dict[str, dict[str, Any]] = {+ Compare.PREVIOUS_PERIOD: {"compare": True},+ Compare.PREVIOUS_DAY: {"compare": True, "compare_to": "-1d"},+ Compare.PREVIOUS_WEEK: {"compare": True, "compare_to": "-7d"},+}+_ORDER_FIELD = {+ OrderBy.ERROR_RATE: "error_rate_pct",+ OrderBy.P95: "p95_ms",+ OrderBy.THROUGHPUT: "requests",+ OrderBy.ERROR_COUNT: "error_count",+}+++Window = tuple[dt.datetime, dt.datetime]+++@frozen+class ServiceWindowStats:+ requests: int+ errors: int+ p50_ms: float+ p95_ms: float+ p99_ms: float+ last_seen: dt.datetime++ @property+ def error_rate_pct(self) -> float:+ return self.errors / self.requests * 100 if self.requests else 0.0+++def _iso(value: dt.datetime) -> str:+ return value.astimezone(UTC).strftime("%Y-%m-%dT%H:%M:%SZ")+++def _range(window: Window) -> dict[str, str]:+ return {"date_from": _iso(window[0]), "date_to": _iso(window[1])}+++def _ms(value: float) -> str:+ return f"{value / 1000:.1f}s" if value >= 1000 else f"{value:.0f}ms"+++def _clip(text: str) -> str:+ return text if len(text) <= MAX_REASON_LENGTH else text[: MAX_REASON_LENGTH - 1] + "…"+++def _round(value: float | None) -> float | None:+ return None if value is None else round(value, 1)+++def _since(value: dt.datetime, reference: dt.datetime) -> str:+ return value.strftime("%H:%MZ" if value.date() == reference.date() else "%Y-%m-%d %H:%MZ")+++def _p95_change_pct(current: ServiceWindowStats, baseline: ServiceWindowStats) -> float | None:+ return (current.p95_ms - baseline.p95_ms) / baseline.p95_ms * 100 if baseline.p95_ms else None+++def classify_service(+ *,+ current: ServiceWindowStats | None,+ baseline: ServiceWindowStats | None,+ has_baseline: bool,+ window_from: dt.datetime,+ window_to: dt.datetime,+ baseline_minutes: float | None = None,+) -> tuple[Status, str]:+ if current is None or not current.requests:+ quiet_for = window_to - baseline.last_seen if baseline is not None else dt.timedelta(0)+ minutes = baseline_minutes or max((window_to - window_from).total_seconds() / 60, 1.0)+ expected = baseline.requests / minutes * SILENT_GAP.total_seconds() / 60 if baseline is not None else 0.0+ if (+ baseline is not None+ and baseline.requests >= SILENT_MIN_BASELINE_REQUESTS+ and quiet_for >= SILENT_GAP+ and expected >= SILENT_MIN_EXPECTED_IN_GAP+ ):+ since = _since(baseline.last_seen, window_to)+ return Status.SILENT, f"no spans since {since} (baseline {baseline.requests} req)"+ return Status.OK, f"no spans; baseline {baseline.requests if baseline else 0} req is too few to call silent"++ gap = window_to - current.last_seen+ rate = current.requests / max((current.last_seen - window_from).total_seconds() / 60, 1.0)+ if gap >= SILENT_GAP and rate * gap.total_seconds() / 60 >= SILENT_MIN_EXPECTED_IN_GAP:+ return Status.SILENT, f"no spans since {_since(current.last_seen, window_to)} (was {rate:.1f}/min)"+ if has_baseline and (baseline is None or not baseline.requests):+ return Status.NEW, "no spans in baseline window"++ rate_now = current.error_rate_pct+ if baseline is not None and min(current.requests, baseline.requests) >= COMPARE_MIN_REQUESTS:+ rate_before = baseline.error_rate_pct+ p95_change = _p95_change_pct(current, baseline)+ errors_up = rate_now - rate_before >= ERROR_RATE_DELTA_PP and rate_now >= ERROR_RATE_RATIO * rate_before+ parts = [f"error rate {rate_before:.1f}% → {rate_now:.1f}%"] if errors_up else []+ if p95_change is not None and p95_change >= P95_RISE_PCT:+ parts.append(f"p95 {_ms(baseline.p95_ms)} → {_ms(current.p95_ms)} (+{p95_change:.0f}%)")+ if parts:+ return Status.REGRESSED, _clip("; ".join(parts))+ if rate_before - rate_now >= ERROR_RATE_DELTA_PP and rate_before >= ERROR_RATE_RATIO * rate_now:+ return Status.RECOVERED, f"error rate {rate_before:.1f}% → {rate_now:.1f}%"+ if rate_now < DEGRADED_ERROR_RATE_PCT and baseline.p95_ms >= current.p95_ms * (1 + P95_RISE_PCT / 100) > 0:+ return Status.IMPROVED, f"p95 {_ms(baseline.p95_ms)} → {_ms(current.p95_ms)} ({p95_change:.0f}%)"+ if current.requests >= DEGRADED_MIN_REQUESTS and rate_now >= DEGRADED_ERROR_RATE_PCT:+ before = f" (baseline {baseline.error_rate_pct:.1f}%)" if baseline is not None and baseline.requests else ""+ return Status.DEGRADED, f"error rate {rate_now:.1f}%{before}"+ return Status.OK, ""+++def aggregate_compare_filter(compare: Compare) -> dict[str, Any] | None:+ return _AGGREGATE_COMPARE_FILTER.get(compare)+++class ServiceHealthQueryRunner:+ def __init__(self, *, team: "Team", params: ServiceHealthParams) -> None:+ self.team = team+ self.params = params+ self.modifiers = create_default_modifiers_for_team(team)+ self.modifiers.convertToProjectTimezone = False+ self.modifiers.propertyGroupsMode = PropertyGroupsMode.OPTIMIZED+ self.current = self._resolve(params.date_from, params.date_to)+ self.baseline = self._baseline_window()++ def _resolve(self, date_from: str | None, date_to: str | None) -> Window:+ query_date_range = QueryDateRange(+ date_range=DateRange(date_from=date_from, date_to=date_to, explicitDate=True),+ team=self.team,+ interval=IntervalType.MINUTE,+ now=dt.datetime.now(),+ timezone_info=UTC,+ exact_timerange=True,+ )+ return query_date_range.date_from().astimezone(UTC), query_date_range.date_to().astimezone(UTC)++ def _baseline_window(self) -> Window | None:+ compare, (start, end) = self.params.compare, self.current+ if compare == Compare.NONE:+ return None+ if compare == Compare.CUSTOM:+ return self._resolve(self.params.compare_date_from, self.params.compare_date_to)+ shift = _SHIFT.get(compare, end - start)+ return start - shift, end - shift++ def window(self) -> dict[str, Any]:+ baseline = _range(self.baseline) if self.baseline else None+ return {"current": _range(self.current), "baseline": baseline, "timezone": "UTC"}++ def byte_cap_message(self) -> str:+ seconds = max(int((self.current[1] - self.current[0]).total_seconds() / 2), 1)+ window = (+ f"-{seconds // 3600}h"+ if seconds % 3600 == 0+ else f"-{seconds // 60}m"+ if seconds % 60 == 0+ else f"-{seconds}s"+ )+ baseline = " and a shorter baselineDateRange" if self.params.compare == Compare.CUSTOM else ""+ return (+ f'This query read more than the 10 GB cap. Retry with a shorter dateRange, at most {{"date_from": "{window}"}}{baseline}, '+ "or pass serviceNames to read fewer services."+ )++ def _execute(self, sql: str, window: Window, service_names: list[str], **placeholders: ast.Expr) -> list:+ dates: dict[str, ast.Expr] = {+ "date_from": ast.Constant(value=window[0]),+ "date_to": ast.Constant(value=window[1]),+ }+ where = [+ parse_expr(TIME_BUCKET_DATE_RANGE_WHERE, placeholders=dates),+ parse_expr("timestamp >= {date_from} AND timestamp < {date_to}", placeholders=dates),+ ]+ if self.params.span_scope == ServiceHealthSpanScope.ENTRY:+ where.append(parse_expr(_ENTRY_WHERE))+ if service_names:+ names = ast.Tuple(exprs=[ast.Constant(value=name) for name in service_names])+ where.append(parse_expr("service_name IN {names}", placeholders={"names": names}))+ if self.params.environment:+ environment = ast.Constant(value=self.params.environment.lower())+ where.append(parse_expr(_ENVIRONMENT_WHERE, placeholders={"env": environment}))+ response = execute_hogql_query(+ query_type="TraceSpansServiceHealthQuery",+ query=parse_select(sql, placeholders={"where": ast.And(exprs=where), **placeholders}),+ team=self.team,+ modifiers=self.modifiers,+ workload=Workload.LOGS,+ limit_context=LimitContext.QUERY,+ settings=fail_fast_scalar_settings(),+ )+ return response.results or []++ def _service_stats(self, window: Window) -> dict[str, ServiceWindowStats]:+ rows = self._execute(+ """+ SELECT service_name, count() AS requests, countIf(status_code = 2),+ quantiles(0.5, 0.95, 0.99)(duration_nano), max(timestamp)+ FROM posthog.trace_spans+ WHERE {where}+ GROUP BY service_name+ ORDER BY requests DESC, service_name ASC+ LIMIT {limit}+ """,+ window,+ list(self.params.service_names),+ limit=ast.Constant(value=SERVICE_SCAN_LIMIT + 1),+ )+ return {+ service: ServiceWindowStats(+ requests=requests,+ errors=errors,+ p50_ms=(quantiles[0] or 0) / 1e6,+ p95_ms=(quantiles[1] or 0) / 1e6,+ p99_ms=(quantiles[2] or 0) / 1e6,+ last_seen=last_seen.replace(tzinfo=UTC) if last_seen.tzinfo is None else last_seen,+ )+ for service, requests, errors, quantiles, last_seen in rows+ }++ def _row(+ self, service: str, current: ServiceWindowStats | None, baseline: ServiceWindowStats | None, cut: bool = False+ ) -> dict:+ status, reason = classify_service(+ current=current,+ baseline=baseline,+ has_baseline=self.baseline is not None and not cut,+ window_from=self.current[0],+ window_to=self.current[1],+ baseline_minutes=(self.baseline[1] - self.baseline[0]).total_seconds() / 60 if self.baseline else None,+ )+ live = current if current is not None and current.requests else None+ delta_pp = p95_change = None+ if live is not None and baseline is not None and baseline.requests:+ delta_pp = _round(live.error_rate_pct - baseline.error_rate_pct)+ p95_change = _round(_p95_change_pct(live, baseline))+ last_seen = live.last_seen if live else (baseline.last_seen if baseline else None)+ minutes = max((self.current[1] - self.current[0]).total_seconds() / 60, 1 / 60)+ return {+ "service": service,+ "status": status,+ "reason": reason,+ "requests": live.requests if live else 0,+ "throughput_per_min": round((live.requests if live else 0) / minutes, 1),+ "error_count": live.errors if live else 0,+ "error_rate_pct": _round(live.error_rate_pct) if live else None,+ "error_rate_delta_pp": delta_pp,+ "p50_ms": _round(live.p50_ms) if live else None,+ "p95_ms": _round(live.p95_ms) if live else None,+ "p95_change_pct": p95_change,+ "p99_ms": _round(live.p99_ms) if live else None,+ "baseline_requests": (baseline.requests if baseline else 0) if self.baseline else None,+ "last_seen": _iso(last_seen) if last_seen else None,+ }++ def _next_calls(self, rows: list[dict]) -> list[dict[str, Any]]:+ unhealthy = [row for row in rows if row["status"] in _UNHEALTHY]+ if self.params.environment or not unhealthy:+ return []+ names = [row["service"] for row in unhealthy[:NEXT_SERVICES_LIMIT]]+ query: dict[str, Any] = {"dateRange": _range(self.current), "serviceNames": names, "limit": 20}+ if compare_filter := aggregate_compare_filter(self.params.compare):+ query["compareFilter"] = compare_filter+ calls = [{"tool": "apm-spans-aggregate", "why": _clip(f"operations behind {', '.join(names)}"), "query": query}]+ worst = next((row for row in unhealthy if row["error_count"]), None)+ if worst:+ query = {"dateRange": _range(self.current), "serviceNames": [worst["service"]], "statusCodes": [2]}+ query = {**query, "flatSpans": True, "rootSpans": False, "limit": 10, "excludeAttributes": True}+ calls.append(+ {"tool": "query-apm-spans", "why": _clip(f"recent error spans in {worst['service']}"), "query": query}+ )+ return calls++ def _notes(self, rows: list[dict], scan_truncated: bool) -> list[str]:+ notes = {+ "No baseline: set compare to get new, regressed, recovered and improved statuses.": self.baseline is None,+ f"Only the {SERVICE_SCAN_LIMIT} busiest services were scored. Pass serviceNames for others.": scan_truncated,+ "No spans matched. spanScope=all also counts internal and client spans. The environment filter reads "+ "deployment.environment.name, deployment.environment, then env.": not rows,+ }+ return [note for note, applies in notes.items() if applies]++ def run(self) -> dict[str, Any]:+ windows = [self.current] if self.baseline is None else [self.current, self.baseline]+ stats = map_in_caller_context(self._service_stats, windows, thread_name_prefix="tracing_service_health")+ current, baseline = stats[0], stats[1] if len(stats) > 1 else {}+ current_cut, baseline_cut = len(current) > SERVICE_SCAN_LIMIT, len(baseline) > SERVICE_SCAN_LIMIT+ names = [*current, *([] if current_cut else sorted(set(baseline) - set(current)))]+ rows = [+ self._row(name, current.get(name), baseline.get(name), baseline_cut and name not in baseline)+ for name in names+ ]++ rows.sort(+ key=lambda row: (+ _STATUS_ORDER.index(row["status"]),+ -row["requests"],+ -(row["baseline_requests"] or 0),+ row["service"],+ )+ )+ field = _ORDER_FIELD.get(self.params.order_by)+ ordered = sorted(rows, key=lambda row: (row[field] is None, -(row[field] or 0))) if field else rows+ page = ordered[: self.params.limit]+ scan_truncated = max(len(current), len(baseline)) > SERVICE_SCAN_LIMIT+ result: dict[str, Any] = {+ "window": self.window(),+ "summary": {+ "services": len(rows),+ "requests": sum(row["requests"] for row in rows),+ **{status.value: sum(row["status"] == status for row in rows) for status in Status},+ },+ "services": page,+ "next": self._next_calls(rows),+ "truncated": len(rows) > self.params.limit or scan_truncated,+ "notes": self._notes(rows, scan_truncated),+ }+ return result+++def run_service_health_query(*, team: "Team", params: ServiceHealthParams) -> dict[str, Any]:+ return ServiceHealthQueryRunner(team=team, params=params).run()+++def service_health_byte_cap_message(*, team: "Team", params: ServiceHealthParams) -> str:+ return ServiceHealthQueryRunner(team=team, params=params).byte_cap_message()diff --git a/products/tracing/backend/tests/test_date_window.py b/products/tracing/backend/tests/test_date_window.pyindex 8d553c1f36de..21cec7b093bf 100644--- a/products/tracing/backend/tests/test_date_window.py+++ b/products/tracing/backend/tests/test_date_window.py@@ -61,6 +61,10 @@ class TestNormalizeTracingWindow:("h",),("-m",),("d",),+ ("-999999999d",),+ ("-99999999999y",),+ (5,),+ (True,),])def test_unparseable_values_raise(self, value: str) -> None:diff --git a/products/tracing/backend/tests/test_get_date_params.py b/products/tracing/backend/tests/test_get_date_params.pynew file mode 100644index 000000000000..c5ced7f21215--- /dev/null+++ b/products/tracing/backend/tests/test_get_date_params.py@@ -0,0 +1,83 @@+import json++from posthog.test.base import APIBaseTest+from unittest.mock import patch++from parameterized import parameterized++VIEWS = "products.tracing.backend.presentation.views"++ENDPOINTS: list[tuple[str, str, dict[str, str], object]] = [+ ("service-names", "run_service_names_query", {}, []),+ ("attributes", "run_attribute_names_query", {}, ([], 0)),+ ("values", "run_attribute_values_query", {"key": "http.method"}, []),+]+++class TestTracingGetDateParams(APIBaseTest):+ def _get(self, path: str, runner: str, params: dict, return_value):+ with patch(f"{VIEWS}.{runner}", return_value=return_value) as run:+ response = self.client.get(f"/api/projects/{self.team.id}/tracing/spans/{path}/", params)+ return response, run++ @parameterized.expand(ENDPOINTS)+ def test_malformed_date_range_is_a_400_with_an_example(self, path, runner, extra, return_value):+ response, run = self._get(path, runner, {"dateRange": "{date_from: -1h}", **extra}, return_value)+ self.assertEqual(response.status_code, 400, response.content)+ self.assertIn('{"date_from": "-1h"}', json.dumps(response.json()).replace('\\"', '"'))+ run.assert_not_called()++ @parameterized.expand(ENDPOINTS)+ def test_date_range_must_be_an_object(self, path, runner, extra, return_value):+ response, run = self._get(path, runner, {"dateRange": '"-1h"', **extra}, return_value)+ self.assertEqual(response.status_code, 400, response.content)+ run.assert_not_called()++ @parameterized.expand(ENDPOINTS)+ def test_date_range_values_must_be_strings(self, path, runner, extra, return_value):+ response, run = self._get(path, runner, {"dateRange": '{"date_from": 5}', **extra}, return_value)+ self.assertEqual(response.status_code, 400, response.content)+ run.assert_not_called()++ @parameterized.expand(ENDPOINTS)+ def test_explicit_date_option_is_still_accepted(self, path, runner, extra, return_value):+ response, run = self._get(+ path, runner, {"dateRange": '{"date_from": "-6h", "explicitDate": true}', **extra}, return_value+ )+ self.assertEqual(response.status_code, 200, response.content)+ self.assertEqual(run.call_args.kwargs["date_range"].date_from, "-6h")++ @parameterized.expand(ENDPOINTS)+ def test_json_date_range_still_works(self, path, runner, extra, return_value):+ response, run = self._get(path, runner, {"dateRange": '{"date_from": "-6h"}', **extra}, return_value)+ self.assertEqual(response.status_code, 200, response.content)+ self.assertEqual(run.call_args.kwargs["date_range"].date_from, "-6h")++ @parameterized.expand(ENDPOINTS)+ def test_flat_date_params(self, path, runner, extra, return_value):+ response, run = self._get(+ path, runner, {"date_from": "-30m", "date_to": "2026-06-02T09:00:00Z", **extra}, return_value+ )+ self.assertEqual(response.status_code, 200, response.content)+ date_range = run.call_args.kwargs["date_range"]+ self.assertEqual((date_range.date_from, date_range.date_to), ("-30M", "2026-06-02T09:00:00Z"))++ @parameterized.expand(ENDPOINTS)+ def test_date_range_wins_over_flat_params(self, path, runner, extra, return_value):+ response, run = self._get(+ path, runner, {"dateRange": '{"date_from": "-6h"}', "date_from": "-2d", **extra}, return_value+ )+ self.assertEqual(response.status_code, 200, response.content)+ self.assertEqual(run.call_args.kwargs["date_range"].date_from, "-6h")++ @parameterized.expand(ENDPOINTS)+ def test_no_dates_defaults_to_the_last_hour(self, path, runner, extra, return_value):+ response, run = self._get(path, runner, extra, return_value)+ self.assertEqual(response.status_code, 200, response.content)+ self.assertEqual(run.call_args.kwargs["date_range"].date_from, "-1h")++ @parameterized.expand(ENDPOINTS)+ def test_bad_flat_value_is_a_400(self, path, runner, extra, return_value):+ response, run = self._get(path, runner, {"date_from": "last tuesday", **extra}, return_value)+ self.assertEqual(response.status_code, 400, response.content)+ run.assert_not_called()diff --git a/products/tracing/backend/tests/test_service_health_endpoint.py b/products/tracing/backend/tests/test_service_health_endpoint.pynew file mode 100644index 000000000000..7534dc0068b5--- /dev/null+++ b/products/tracing/backend/tests/test_service_health_endpoint.py@@ -0,0 +1,211 @@+import datetime as dt++from posthog.test.base import APIBaseTest+from unittest.mock import patch++from parameterized import parameterized++from posthog.clickhouse.client import sync_execute+from posthog.errors import CHQueryErrorTooManyBytes++from products.tracing.backend.facade.contracts import ServiceHealthParams+from products.tracing.backend.facade.enums import ServiceHealthCompare, ServiceHealthOrderBy, ServiceHealthSpanScope+from products.tracing.backend.tests.test_keyset_pagination import _TraceSpansTestBase++RUNNER_PATH = "products.tracing.backend.presentation.views.run_service_health_query"+++def _url(team_id: int) -> str:+ return f"/api/projects/{team_id}/tracing/spans/service_health/"+++class TestServiceHealthValidation(APIBaseTest):+ @parameterized.expand(+ [+ ("missing_query", {}),+ ("query_not_object", {"query": "x"}),+ ("unknown_compare", {"query": {"compare": "yesterday"}}),+ ("compare_object", {"query": {"compare": {"date_from": "-2d"}}}),+ ("baseline_not_object", {"query": {}, "baselineDateRange": "-2d"}),+ ("compare_wrong_type", {"query": {"compare": 3}}),+ ("bad_span_scope", {"query": {"spanScope": "root"}}),+ ("bad_order_by", {"query": {"orderBy": "name"}}),+ ("reversed_baseline", {"query": {}, "baselineDateRange": {"date_from": "-1d", "date_to": "-2d"}}),+ ("future_window", {"query": {"dateRange": {"date_from": "2099-01-01T00:00:00Z"}}}),+ ("future_baseline", {"query": {}, "baselineDateRange": {"date_from": "2099-01-01T00:00:00Z"}}),+ ("limit_too_small", {"query": {"limit": 0}}),+ ("limit_too_large", {"query": {"limit": 1000}}),+ ("bad_date", {"query": {"dateRange": {"date_from": "yesterday-ish"}}}),+ ("date_range_not_object", {"query": {"dateRange": "-1h"}}),+ ("numeric_date", {"query": {"dateRange": {"date_from": 5}}}),+ ("huge_window", {"query": {"dateRange": {"date_from": "-999999999d"}}}),+ (+ "year_one",+ {"query": {"dateRange": {"date_from": "0001-01-03T00:00:00Z", "date_to": "0001-01-10T00:00:00Z"}}},+ ),+ ]+ )+ def test_rejects_bad_input(self, _name, body):+ with patch(RUNNER_PATH) as runner:+ response = self.client.post(_url(self.team.id), body, format="json")+ self.assertEqual(response.status_code, 400, response.content)+ runner.assert_not_called()++ def test_defaults(self):+ with patch(RUNNER_PATH, side_effect=CHQueryErrorTooManyBytes("too many bytes")) as runner:+ self.client.post(_url(self.team.id), {"query": {}}, format="json")+ params = runner.call_args.kwargs["params"]+ self.assertEqual(+ params,+ ServiceHealthParams(+ date_from="-1h",+ date_to=None,+ compare=ServiceHealthCompare.PREVIOUS_PERIOD,+ environment=None,+ service_names=(),+ span_scope=ServiceHealthSpanScope.ENTRY,+ order_by=ServiceHealthOrderBy.STATUS,+ limit=25,+ ),+ )++ def test_parses_every_field(self):+ body = {+ "query": {+ "dateRange": {"date_from": "-30m", "date_to": None},+ "environment": "production",+ "serviceNames": ["api"],+ "spanScope": "all",+ "orderBy": "p95",+ "limit": 5,+ },+ "baselineDateRange": {"date_from": "-2d", "date_to": "-1d"},+ }+ with patch(RUNNER_PATH, side_effect=CHQueryErrorTooManyBytes("too many bytes")) as runner:+ self.client.post(_url(self.team.id), body, format="json")+ params = runner.call_args.kwargs["params"]+ self.assertEqual(params.date_from, "-30M")+ self.assertEqual(params.compare, ServiceHealthCompare.CUSTOM)+ self.assertEqual((params.compare_date_from, params.compare_date_to), ("-2d", "-1d"))+ self.assertEqual(params.service_names, ("api",))+ self.assertEqual((params.span_scope, params.order_by, params.limit), ("all", "p95", 5))++ @parameterized.expand(["none", "previous_period", "previous_day", "previous_week"])+ def test_accepts_compare_names(self, compare):+ with patch(RUNNER_PATH, side_effect=CHQueryErrorTooManyBytes("too many bytes")) as runner:+ self.client.post(_url(self.team.id), {"query": {"compare": compare}}, format="json")+ self.assertEqual(runner.call_args.kwargs["params"].compare, compare)++ def test_byte_cap_returns_an_actionable_400(self):+ with patch(RUNNER_PATH, side_effect=CHQueryErrorTooManyBytes("too many bytes")):+ response = self.client.post(+ _url(self.team.id), {"query": {"dateRange": {"date_from": "-7d"}}}, format="json"+ )+ self.assertEqual(response.status_code, 400, response.content)+ detail = response.json()["detail"]+ self.assertIn("10 GB", detail)+ self.assertIn("serviceNames", detail)+ self.assertIn('{"date_from": "-84h"}', detail)++ def test_byte_cap_on_a_short_window_recommends_a_smaller_one(self):+ with patch(RUNNER_PATH, side_effect=CHQueryErrorTooManyBytes("too many bytes")):+ response = self.client.post(+ _url(self.team.id),+ {"query": {"dateRange": {"date_from": "-1h"}}, "baselineDateRange": {"date_from": "-3h"}},+ format="json",+ )+ detail = response.json()["detail"]+ self.assertIn('{"date_from": "-30m"}', detail)+ self.assertIn("baselineDateRange", detail)++ def test_byte_cap_on_a_one_minute_window_recommends_seconds(self):+ with patch(RUNNER_PATH, side_effect=CHQueryErrorTooManyBytes("too many bytes")):+ response = self.client.post(+ _url(self.team.id), {"query": {"dateRange": {"date_from": "-1m"}}}, format="json"+ )+ self.assertIn('{"date_from": "-30s"}', response.json()["detail"])+++class TestServiceHealthResponse(_TraceSpansTestBase):+ @classmethod+ def setUpTestData(cls):+ super().setUpTestData()+ cls._recreate_trace_spans_tables()+ rows: list[tuple] = []+ for i in range(120):+ for service, start, errors in (+ ("worker", dt.datetime(2026, 6, 2, 7, 0), 0),+ ("worker", dt.datetime(2026, 6, 2, 8, 0), 12),+ ):+ ts = start + dt.timedelta(seconds=30 * i)+ rows.append(+ (+ f"019e8810-0000-0000-0000-{len(rows):012d}",+ cls.team.id,+ f"trace-{len(rows)}",+ f"span-{len(rows)}",+ "",+ "consume",+ 5,+ ts,+ ts + dt.timedelta(milliseconds=20),+ ts,+ 2 if i < errors else 0,+ service,+ {},+ )+ )+ sync_execute(+ "INSERT INTO trace_spans (uuid, team_id, trace_id, span_id, parent_span_id, name, kind, timestamp, "+ "end_time, observed_timestamp, status_code, service_name, resource_attributes) VALUES",+ rows,+ )++ def _post(self, **query) -> dict:+ body = {+ "query": {"dateRange": {"date_from": "2026-06-02T08:00:00Z", "date_to": "2026-06-02T09:00:00Z"}, **query}+ }+ response = self.client.post(_url(self.team.id), body, format="json")+ self.assertEqual(response.status_code, 200, response.content)+ return response.json()++ def test_response_shape(self):+ data = self._post()+ self.assertEqual(set(data), {"window", "summary", "services", "next", "truncated", "notes"})+ self.assertEqual(+ data["window"],+ {+ "current": {"date_from": "2026-06-02T08:00:00Z", "date_to": "2026-06-02T09:00:00Z"},+ "baseline": {"date_from": "2026-06-02T07:00:00Z", "date_to": "2026-06-02T08:00:00Z"},+ "timezone": "UTC",+ },+ )+ self.assertEqual(data["summary"]["services"], 1)+ self.assertEqual(data["summary"]["regressed"], 1)+ [row] = data["services"]+ self.assertEqual(+ list(row),+ [+ "service",+ "status",+ "reason",+ "requests",+ "throughput_per_min",+ "error_count",+ "error_rate_pct",+ "error_rate_delta_pp",+ "p50_ms",+ "p95_ms",+ "p95_change_pct",+ "p99_ms",+ "baseline_requests",+ "last_seen",+ ],+ )+ self.assertEqual(row["service"], "worker")+ self.assertEqual(row["status"], "regressed")+ self.assertEqual(row["error_rate_pct"], 10.0)+ self.assertEqual(row["p95_ms"], 20.0)+ self.assertTrue(row["last_seen"].endswith("Z"))+ self.assertFalse(data["truncated"])+ self.assertEqual(data["next"][0]["tool"], "apm-spans-aggregate")diff --git a/products/tracing/backend/tests/test_service_health_query_runner.py b/products/tracing/backend/tests/test_service_health_query_runner.pynew file mode 100644index 000000000000..5c3a7473afc3--- /dev/null+++ b/products/tracing/backend/tests/test_service_health_query_runner.py@@ -0,0 +1,428 @@+import datetime as dt++import time_machine+from posthog.test.base import BaseTest+from unittest.mock import patch++from django.test import SimpleTestCase++from parameterized import parameterized++from posthog.clickhouse.client import sync_execute++from products.tracing.backend.facade.contracts import ServiceHealthParams+from products.tracing.backend.facade.enums import (+ ServiceHealthCompare,+ ServiceHealthOrderBy,+ ServiceHealthSpanScope,+ ServiceHealthStatus,+)+from products.tracing.backend.service_health_query_runner import (+ MAX_REASON_LENGTH,+ ServiceHealthQueryRunner,+ ServiceWindowStats,+ aggregate_compare_filter,+ classify_service,+)+from products.tracing.backend.tests.test_keyset_pagination import _TraceSpansTestBase++CURRENT_FROM = dt.datetime(2026, 6, 2, 8, 0, 0)+CURRENT_TO = dt.datetime(2026, 6, 2, 9, 0, 0)+BASELINE_FROM = dt.datetime(2026, 6, 2, 7, 0, 0)+PREVIOUS_DAY_FROM = dt.datetime(2026, 6, 1, 8, 0, 0)++PRODUCTION = {"deployment.environment.name": "production"}+++def _stats(+ requests: int,+ errors: int = 0,+ p95_ms: float = 10.0,+ last_seen: dt.datetime | None = None,+) -> ServiceWindowStats:+ return ServiceWindowStats(+ requests=requests,+ errors=errors,+ p50_ms=p95_ms,+ p95_ms=p95_ms,+ p99_ms=p95_ms,+ last_seen=last_seen or CURRENT_TO - dt.timedelta(seconds=20),+ )+++class TestClassifyService(SimpleTestCase):+ @parameterized.expand(+ [+ ("ok", _stats(200), _stats(200), ServiceHealthStatus.OK),+ (+ "silent_no_current_spans",+ None,+ _stats(100, last_seen=CURRENT_FROM - dt.timedelta(minutes=10)),+ ServiceHealthStatus.SILENT,+ ),+ ("small_baseline_gone_is_not_silent", None, _stats(5), ServiceHealthStatus.OK),+ (+ "silent_spans_stopped_inside_window",+ _stats(300, last_seen=CURRENT_FROM + dt.timedelta(minutes=30)),+ _stats(300),+ ServiceHealthStatus.SILENT,+ ),+ ("new", _stats(50), None, ServiceHealthStatus.NEW),+ ("regressed_error_rate", _stats(200, errors=10), _stats(200, errors=1), ServiceHealthStatus.REGRESSED),+ ("regressed_p95", _stats(200, p95_ms=30.0), _stats(200, p95_ms=10.0), ServiceHealthStatus.REGRESSED),+ ("error_rise_below_ratio", _stats(200, errors=24), _stats(200, errors=20), ServiceHealthStatus.DEGRADED),+ ("regression_needs_volume", _stats(50, errors=10), _stats(50), ServiceHealthStatus.DEGRADED),+ ("degraded", _stats(200, errors=20), _stats(200, errors=20), ServiceHealthStatus.DEGRADED),+ ("recovered", _stats(200), _stats(200, errors=20), ServiceHealthStatus.RECOVERED),+ ("improved", _stats(200, p95_ms=10.0), _stats(200, p95_ms=40.0), ServiceHealthStatus.IMPROVED),+ ]+ )+ def test_status_rules(self, _name, current, baseline, expected):+ status, reason = classify_service(+ current=current, baseline=baseline, has_baseline=True, window_from=CURRENT_FROM, window_to=CURRENT_TO+ )+ self.assertEqual(status, expected, reason)+ self.assertLessEqual(len(reason), MAX_REASON_LENGTH)++ def test_regression_reason_names_the_rates(self):+ _, reason = classify_service(+ current=_stats(200, errors=10),+ baseline=_stats(200, errors=1),+ has_baseline=True,+ window_from=CURRENT_FROM,+ window_to=CURRENT_TO,+ )+ self.assertEqual(reason, "error rate 0.5% → 5.0%")++ def test_silent_reason_names_the_last_span(self):+ _, reason = classify_service(+ current=_stats(300, last_seen=dt.datetime(2026, 6, 2, 8, 30, 0)),+ baseline=None,+ has_baseline=False,+ window_from=CURRENT_FROM,+ window_to=CURRENT_TO,+ )+ self.assertTrue(reason.startswith("no spans since 08:30Z"), reason)++ def test_sparse_baseline_is_not_silent(self):+ status, _ = classify_service(+ current=None,+ baseline=_stats(30, last_seen=CURRENT_FROM - dt.timedelta(minutes=30)),+ has_baseline=True,+ window_from=CURRENT_FROM,+ window_to=CURRENT_TO,+ baseline_minutes=60,+ )+ self.assertNotEqual(status, ServiceHealthStatus.SILENT)++ def test_no_current_spans_is_not_silent_inside_the_gap(self):+ status, _ = classify_service(+ current=None,+ baseline=_stats(300, last_seen=CURRENT_TO - dt.timedelta(minutes=5)),+ has_baseline=True,+ window_from=CURRENT_TO - dt.timedelta(minutes=5),+ window_to=CURRENT_TO,+ )+ self.assertNotEqual(status, ServiceHealthStatus.SILENT)++ def test_without_a_baseline_only_absolute_rules_apply(self):+ status, _ = classify_service(+ current=_stats(200, errors=10),+ baseline=None,+ has_baseline=False,+ window_from=CURRENT_FROM,+ window_to=CURRENT_TO,+ )+ self.assertEqual(status, ServiceHealthStatus.DEGRADED)+ status, _ = classify_service(+ current=_stats(200), baseline=None, has_baseline=False, window_from=CURRENT_FROM, window_to=CURRENT_TO+ )+ self.assertEqual(status, ServiceHealthStatus.OK)+++class TestAggregateCompareFilter(SimpleTestCase):+ @parameterized.expand(+ [+ (ServiceHealthCompare.PREVIOUS_PERIOD, {"compare": True}),+ (ServiceHealthCompare.PREVIOUS_DAY, {"compare": True, "compare_to": "-1d"}),+ (ServiceHealthCompare.PREVIOUS_WEEK, {"compare": True, "compare_to": "-7d"}),+ (ServiceHealthCompare.CUSTOM, None),+ (ServiceHealthCompare.NONE, None),+ ]+ )+ def test_maps_compare_to_the_aggregate_tool(self, compare, expected):+ self.assertEqual(aggregate_compare_filter(compare), expected)+++class TestServiceHealthWindows(BaseTest):+ @parameterized.expand(+ [+ (ServiceHealthCompare.PREVIOUS_PERIOD, "2026-06-02T06:00:00Z", "2026-06-02T08:00:00Z"),+ (ServiceHealthCompare.PREVIOUS_DAY, "2026-06-01T08:00:00Z", "2026-06-01T10:00:00Z"),+ (ServiceHealthCompare.PREVIOUS_WEEK, "2026-05-26T08:00:00Z", "2026-05-26T10:00:00Z"),+ ]+ )+ @time_machine.travel("2026-06-02T10:00:00Z", tick=False)+ def test_baseline_window(self, compare, expected_from, expected_to):+ runner = ServiceHealthQueryRunner(team=self.team, params=ServiceHealthParams(date_from="-2h", compare=compare))+ window = runner.window()+ self.assertEqual(window["current"], {"date_from": "2026-06-02T08:00:00Z", "date_to": "2026-06-02T10:00:00Z"})+ self.assertEqual(window["baseline"], {"date_from": expected_from, "date_to": expected_to})+ self.assertEqual(window["timezone"], "UTC")++ def test_custom_baseline_window(self):+ params = ServiceHealthParams(+ date_from="2026-06-02T08:00:00Z",+ date_to="2026-06-02T09:00:00Z",+ compare=ServiceHealthCompare.CUSTOM,+ compare_date_from="2026-05-30T00:00:00Z",+ compare_date_to="2026-05-30T06:00:00Z",+ )+ window = ServiceHealthQueryRunner(team=self.team, params=params).window()+ self.assertEqual(window["baseline"], {"date_from": "2026-05-30T00:00:00Z", "date_to": "2026-05-30T06:00:00Z"})++ def test_no_baseline(self):+ params = ServiceHealthParams(date_from="-1h", compare=ServiceHealthCompare.NONE)+ self.assertIsNone(ServiceHealthQueryRunner(team=self.team, params=params).window()["baseline"])+++def _spread(start: dt.datetime, count: int, minutes: int = 60) -> list[dt.datetime]:+ step = dt.timedelta(seconds=minutes * 60 / count)+ return [start + step * i for i in range(count)]+++class TestServiceHealthQueryRunner(_TraceSpansTestBase):+ @classmethod+ def setUpTestData(cls):+ super().setUpTestData()+ cls._recreate_trace_spans_tables()+ rows: list[tuple] = []++ def add(+ service: str,+ start: dt.datetime,+ count: int,+ *,+ errors: int = 0,+ duration_ms: float = 10.0,+ name: str = "GET /",+ kind: int = 2,+ parent: str = "",+ resource: dict | None = None,+ minutes: int = 60,+ ) -> None:+ for i, ts in enumerate(_spread(start, count, minutes)):+ index = len(rows)+ rows.append(+ (+ f"019e8800-0000-0000-0000-{index:012d}",+ cls.team.id,+ f"trace-{index}",+ f"span-{index}",+ parent,+ name,+ kind,+ ts,+ ts + dt.timedelta(milliseconds=duration_ms),+ ts,+ 2 if i < errors else 0,+ service,+ PRODUCTION if resource is None else resource,+ )+ )++ add("api", BASELINE_FROM, 200)+ add("api", CURRENT_FROM, 200)+ add("api", CURRENT_FROM, 200, errors=200, kind=1, parent="cGFyZW50", name="db query")+ add("api", CURRENT_FROM, 40, resource={"deployment.environment": "staging"})+ add("stage-svc", CURRENT_FROM, 30, resource={"env": "staging"})+ add("worker", BASELINE_FROM, 200, errors=1, kind=5, parent="cGFyZW50", name="consume order")+ add("worker", CURRENT_FROM, 150, errors=10, kind=5, parent="cGFyZW50", name="consume order")+ add("worker", CURRENT_FROM, 50, errors=2, kind=5, parent="cGFyZW50", name="consume refund")+ add("dispatch", BASELINE_FROM, 100)+ add("billing", CURRENT_FROM, 50)+ add("search", BASELINE_FROM, 200, duration_ms=10.0)+ add("search", CURRENT_FROM, 200, duration_ms=30.0)+ add("mailer", BASELINE_FROM, 200, errors=20)+ add("mailer", CURRENT_FROM, 200, errors=20)+ add("ledger", BASELINE_FROM, 200, errors=20)+ add("ledger", CURRENT_FROM, 200)+ add("cache", BASELINE_FROM, 200, duration_ms=40.0)+ add("cache", CURRENT_FROM, 200, duration_ms=10.0)+ add("cron", BASELINE_FROM, 150)+ add("cron", CURRENT_FROM, 150, minutes=30)+ add("nightly", PREVIOUS_DAY_FROM, 100)++ sync_execute(+ "INSERT INTO trace_spans (uuid, team_id, trace_id, span_id, parent_span_id, name, kind, timestamp, "+ "end_time, observed_timestamp, status_code, service_name, resource_attributes) VALUES",+ rows,+ )++ def _run(self, **overrides):+ fields = {"date_from": "2026-06-02T08:00:00Z", "date_to": "2026-06-02T09:00:00Z", "environment": "production"}+ params = ServiceHealthParams(**{**fields, **overrides})+ return ServiceHealthQueryRunner(team=self.team, params=params).run()++ def _by_service(self, result) -> dict:+ return {row["service"]: row for row in result["services"]}++ def test_window_is_echoed_in_utc(self):+ result = self._run()+ self.assertEqual(result["window"]["current"]["date_from"], "2026-06-02T08:00:00Z")+ self.assertEqual(result["window"]["current"]["date_to"], "2026-06-02T09:00:00Z")+ self.assertEqual(result["window"]["baseline"]["date_from"], "2026-06-02T07:00:00Z")+ self.assertEqual(result["window"]["baseline"]["date_to"], "2026-06-02T08:00:00Z")++ def test_each_status(self):+ rows = self._by_service(self._run(limit=100))+ self.assertEqual(+ {service: row["status"] for service, row in rows.items()},+ {+ "api": ServiceHealthStatus.OK,+ "worker": ServiceHealthStatus.REGRESSED,+ "dispatch": ServiceHealthStatus.SILENT,+ "billing": ServiceHealthStatus.NEW,+ "search": ServiceHealthStatus.REGRESSED,+ "mailer": ServiceHealthStatus.DEGRADED,+ "ledger": ServiceHealthStatus.RECOVERED,+ "cache": ServiceHealthStatus.IMPROVED,+ "cron": ServiceHealthStatus.SILENT,+ },+ )++ def test_row_metrics_use_units(self):+ worker = self._by_service(self._run())["worker"]+ self.assertEqual(worker["requests"], 200)+ self.assertEqual(worker["error_count"], 12)+ self.assertEqual(worker["error_rate_pct"], 6.0)+ self.assertEqual(worker["error_rate_delta_pp"], 5.5)+ self.assertEqual(worker["baseline_requests"], 200)+ self.assertEqual(worker["throughput_per_min"], 3.3)+ self.assertEqual(worker["p95_ms"], 10.0)+ self.assertEqual(worker["p95_change_pct"], 0.0)+ self.assertEqual(worker["reason"], "error rate 0.5% → 6.0%")++ search = self._by_service(self._run())["search"]+ self.assertEqual((search["p50_ms"], search["p95_ms"], search["p99_ms"]), (30.0, 30.0, 30.0))+ self.assertEqual(search["p95_change_pct"], 200.0)++ def test_silent_row_has_no_current_metrics(self):+ dispatch = self._by_service(self._run())["dispatch"]+ self.assertEqual(dispatch["requests"], 0)+ self.assertEqual(dispatch["baseline_requests"], 100)+ self.assertIsNone(dispatch["error_rate_pct"])+ self.assertIsNone(dispatch["p95_ms"])+ self.assertTrue(dispatch["reason"].startswith("no spans since 07:"), dispatch["reason"])++ def test_entry_scope_skips_internal_spans(self):+ entry = self._by_service(self._run())["api"]+ all_spans = self._by_service(self._run(span_scope=ServiceHealthSpanScope.ALL))["api"]+ self.assertEqual((entry["requests"], entry["error_count"]), (200, 0))+ self.assertEqual((all_spans["requests"], all_spans["error_count"]), (400, 200))+ self.assertEqual(all_spans["status"], ServiceHealthStatus.REGRESSED)++ def test_environment_uses_the_fallback_chain(self):+ rows = self._by_service(self._run(environment="Staging"))+ self.assertEqual(set(rows), {"api", "stage-svc"})+ self.assertEqual(rows["api"]["requests"], 40)+ self.assertEqual(rows["stage-svc"]["requests"], 30)++ def test_no_environment_counts_every_label(self):+ params = ServiceHealthParams(date_from="2026-06-02T08:00:00Z", date_to="2026-06-02T09:00:00Z")+ rows = self._by_service(ServiceHealthQueryRunner(team=self.team, params=params).run())+ self.assertEqual(rows["api"]["requests"], 240)++ def test_service_names_filter(self):+ result = self._run(service_names=("worker", "dispatch"))+ self.assertEqual({row["service"] for row in result["services"]}, {"worker", "dispatch"})++ def test_previous_day_compare(self):+ rows = self._by_service(self._run(compare=ServiceHealthCompare.PREVIOUS_DAY, limit=100))+ self.assertEqual(rows["nightly"]["status"], ServiceHealthStatus.SILENT)+ self.assertEqual(rows["api"]["status"], ServiceHealthStatus.NEW)++ def test_custom_compare(self):+ rows = self._by_service(+ self._run(+ compare=ServiceHealthCompare.CUSTOM,+ compare_date_from="2026-06-01T08:00:00Z",+ compare_date_to="2026-06-01T09:00:00Z",+ limit=100,+ )+ )+ self.assertEqual(rows["nightly"]["status"], ServiceHealthStatus.SILENT)++ def test_no_compare(self):+ result = self._run(compare=ServiceHealthCompare.NONE, limit=100)+ rows = self._by_service(result)+ self.assertNotIn("dispatch", rows)+ self.assertEqual(rows["billing"]["status"], ServiceHealthStatus.OK)+ self.assertIsNone(rows["worker"]["error_rate_delta_pp"])+ self.assertIsNone(result["window"]["baseline"])+ self.assertTrue(any("compare" in note for note in result["notes"]))++ def test_order_by_status_puts_the_worst_first(self):+ statuses = [row["status"] for row in self._run(limit=100)["services"]]+ self.assertEqual(statuses[:4], [ServiceHealthStatus.SILENT] * 2 + [ServiceHealthStatus.REGRESSED] * 2)+ self.assertEqual(statuses[-1], ServiceHealthStatus.OK)++ @parameterized.expand(+ [+ (ServiceHealthOrderBy.ERROR_RATE, "mailer"),+ (ServiceHealthOrderBy.ERROR_COUNT, "mailer"),+ (ServiceHealthOrderBy.P95, "search"),+ ]+ )+ def test_order_by_metric(self, order_by, expected_first):+ self.assertEqual(self._run(order_by=order_by, limit=100)["services"][0]["service"], expected_first)++ def test_order_by_throughput(self):+ rows = self._run(order_by=ServiceHealthOrderBy.THROUGHPUT, limit=100)["services"]+ requests = [row["requests"] for row in rows]+ self.assertEqual(requests, sorted(requests, reverse=True))++ def test_limit_truncates_and_summary_counts_everything(self):+ result = self._run(limit=2)+ self.assertEqual(len(result["services"]), 2)+ self.assertTrue(result["truncated"])+ self.assertEqual(result["summary"]["services"], 9)+ self.assertEqual(result["summary"]["silent"], 2)+ self.assertEqual(result["summary"]["regressed"], 2)+ self.assertEqual(result["summary"]["degraded"], 1)+ self.assertEqual(result["summary"]["new"], 1)++ def test_next_points_at_the_drill_down(self):+ result = self._run(environment=None)+ calls = {call["tool"]: call for call in result["next"]}+ self.assertEqual(set(calls), {"apm-spans-aggregate", "query-apm-spans"})+ aggregate = calls["apm-spans-aggregate"]+ self.assertEqual(aggregate["query"]["serviceNames"], ["search", "worker", "mailer"])+ self.assertEqual(+ aggregate["query"]["dateRange"], {"date_from": "2026-06-02T08:00:00Z", "date_to": "2026-06-02T09:00:00Z"}+ )+ self.assertEqual(aggregate["query"]["compareFilter"], {"compare": True})+ error_spans = calls["query-apm-spans"]["query"]+ self.assertEqual((error_spans["serviceNames"], error_spans["statusCodes"]), (["worker"], [2]))+ self.assertEqual((error_spans["flatSpans"], error_spans["rootSpans"]), (True, False))+ for call in result["next"]:+ self.assertLessEqual(len(call["why"]), MAX_REASON_LENGTH)+ self.assertEqual(self._run()["next"], [])++ def test_a_truncated_scan_does_not_invent_silent_or_new_services(self):+ with patch("products.tracing.backend.service_health_query_runner.SERVICE_SCAN_LIMIT", 3):+ result = self._run()+ statuses = {row["status"] for row in result["services"]}+ self.assertFalse(statuses & {ServiceHealthStatus.SILENT, ServiceHealthStatus.NEW}, statuses)+ self.assertTrue(result["truncated"])++ def test_runs_with_fail_fast_settings(self):+ with patch(+ "products.tracing.backend.service_health_query_runner.execute_hogql_query",+ side_effect=lambda **kwargs: type("R", (), {"results": []})(),+ ) as execute:+ self._run()+ settings = execute.call_args.kwargs["settings"]+ self.assertEqual(settings.max_execution_time, 30)+ self.assertEqual(settings.max_bytes_to_read, 10_000_000_000)+ self.assertEqual(settings.read_overflow_mode, "throw")diff --git a/products/tracing/frontend/generated/api.schemas.ts b/products/tracing/frontend/generated/api.schemas.tsindex ad5782e73adc..3923e3a14a74 100644--- a/products/tracing/frontend/generated/api.schemas.ts+++ b/products/tracing/frontend/generated/api.schemas.ts@@ -738,6 +738,262 @@ export interface _TracingServiceNamesResponseApi {results: _TracingServiceNameApi[]}+export interface _ServiceHealthDateRangeApi {+ /**+ * Relative (-30m, -6h, -7d) or ISO 8601.+ * @nullable+ */+ date_from?: string | null+ /**+ * Same format. Omit or null for "now".+ * @nullable+ */+ date_to?: string | null+}++/**+ * * `none` - None+ * * `previous_period` - Previous Period+ * * `previous_day` - Previous Day+ * * `previous_week` - Previous Week+ */+export type ServiceHealthBaselineEnumApi =+ (typeof ServiceHealthBaselineEnumApi)[keyof typeof ServiceHealthBaselineEnumApi]++export const ServiceHealthBaselineEnumApi = {+ None: 'none',+ PreviousPeriod: 'previous_period',+ PreviousDay: 'previous_day',+ PreviousWeek: 'previous_week',+} as const++/**+ * * `entry` - Entry+ * * `all` - All+ */+export type ServiceHealthSpanScopeEnumApi =+ (typeof ServiceHealthSpanScopeEnumApi)[keyof typeof ServiceHealthSpanScopeEnumApi]++export const ServiceHealthSpanScopeEnumApi = {+ Entry: 'entry',+ All: 'all',+} as const++/**+ * * `status` - Status+ * * `error_rate` - Error Rate+ * * `p95` - P95+ * * `throughput` - Throughput+ * * `error_count` - Error Count+ */+export type ServiceHealthOrderByEnumApi = (typeof ServiceHealthOrderByEnumApi)[keyof typeof ServiceHealthOrderByEnumApi]++export const ServiceHealthOrderByEnumApi = {+ Status: 'status',+ ErrorRate: 'error_rate',+ P95: 'p95',+ Throughput: 'throughput',+ ErrorCount: 'error_count',+} as const++export interface _ServiceHealthQueryApi {+ /** The window to score. Defaults to the last hour. */+ dateRange?: _ServiceHealthDateRangeApi+ /** The baseline. previous_period (default) is the same length just before the window. previous_day and previous_week shift the window back 1 or 7 days. none skips it. baselineDateRange overrides it.+ *+ * * `none` - None+ * * `previous_period` - Previous Period+ * * `previous_day` - Previous Day+ * * `previous_week` - Previous Week */+ compare?: ServiceHealthBaselineEnumApi+ /**+ * Only this environment, matched without case on deployment.environment.name, then deployment.environment, then env.+ * @nullable+ */+ environment?: string | null+ /**+ * Only these services.+ * @maxItems 100+ */+ serviceNames?: string[]+ /** entry (default) counts root, server and consumer spans, one per request. all counts every span.+ *+ * * `entry` - Entry+ * * `all` - All */+ spanScope?: ServiceHealthSpanScopeEnumApi+ /** status (default) puts silent, regressed and degraded first. The others sort high to low.+ *+ * * `status` - Status+ * * `error_rate` - Error Rate+ * * `p95` - P95+ * * `throughput` - Throughput+ * * `error_count` - Error Count */+ orderBy?: ServiceHealthOrderByEnumApi+ /**+ * Max service rows, 1-200.+ * @minimum 1+ * @maximum 200+ */+ limit?: number+}++export interface ServiceHealthRequestApi {+ /** The service health query. Every field is optional. */+ query: _ServiceHealthQueryApi+ /** An explicit baseline window. Overrides query.compare. */+ baselineDateRange?: _ServiceHealthDateRangeApi+}++export interface _ServiceHealthTimeRangeApi {+ /** Start, ISO 8601 UTC, inclusive. */+ date_from: string+ /** End, ISO 8601 UTC, exclusive. */+ date_to: string+}++export interface _ServiceHealthWindowApi {+ /** The scored window. Reuse it as dateRange in follow-up calls. */+ current: _ServiceHealthTimeRangeApi+ /** The baseline window. Null when compare is none. */+ baseline: _ServiceHealthTimeRangeApi | null+ /** Always UTC. */+ timezone: string+}++export interface _ServiceHealthSummaryApi {+ /** Services scored, before the limit. */+ services: number+ /** Requests over all scored services. */+ requests: number+ /** Services per status. */+ ok: number+ /** Services per status. */+ degraded: number+ /** Services per status. */+ regressed: number+ /** Services per status. */+ silent: number+ /** Services per status. */+ new: number+ /** Services per status. */+ recovered: number+ /** Services per status. */+ improved: number+}++/**+ * * `silent` - Silent+ * * `regressed` - Regressed+ * * `degraded` - Degraded+ * * `new` - New+ * * `recovered` - Recovered+ * * `improved` - Improved+ * * `ok` - Ok+ */+export type ServiceHealthStatusEnumApi = (typeof ServiceHealthStatusEnumApi)[keyof typeof ServiceHealthStatusEnumApi]++export const ServiceHealthStatusEnumApi = {+ Silent: 'silent',+ Regressed: 'regressed',+ Degraded: 'degraded',+ New: 'new',+ Recovered: 'recovered',+ Improved: 'improved',+ Ok: 'ok',+} as const++export interface _ServiceHealthRowApi {+ /** Service name. */+ service: string+ /** silent: spans stopped. regressed: error rate or p95 worse than the baseline. degraded: error rate 5% or more. new: no spans in the baseline. recovered, improved: the reverse of regressed.+ *+ * * `silent` - Silent+ * * `regressed` - Regressed+ * * `degraded` - Degraded+ * * `new` - New+ * * `recovered` - Recovered+ * * `improved` - Improved+ * * `ok` - Ok */+ status: ServiceHealthStatusEnumApi+ /** Why the service has this status. Empty for ok. */+ reason: string+ /** Spans in the window, in the chosen spanScope. */+ requests: number+ /** Requests per minute. */+ throughput_per_min: number+ /** Requests with OTel status Error. */+ error_count: number+ /**+ * Error share of requests, percent.+ * @nullable+ */+ error_rate_pct: number | null+ /**+ * Change from the baseline, points.+ * @nullable+ */+ error_rate_delta_pp: number | null+ /**+ * Median duration, ms.+ * @nullable+ */+ p50_ms: number | null+ /**+ * 95th percentile duration, ms.+ * @nullable+ */+ p95_ms: number | null+ /**+ * p95 change from the baseline, percent.+ * @nullable+ */+ p95_change_pct: number | null+ /**+ * 99th percentile duration, ms.+ * @nullable+ */+ p99_ms: number | null+ /**+ * Baseline requests. Null without one.+ * @nullable+ */+ baseline_requests: number | null+ /**+ * Last span, ISO 8601 UTC.+ * @nullable+ */+ last_seen: string | null+}++/**+ * The prefilled query argument.+ */+export type _ServiceHealthNextCallApiQuery = { [key: string]: unknown }++export interface _ServiceHealthNextCallApi {+ /** The tool to call next. */+ tool: string+ /** What the call answers. */+ why: string+ /** The prefilled query argument. */+ query: _ServiceHealthNextCallApiQuery+}++export interface ServiceHealthResponseApi {+ /** The absolute UTC windows the server used. */+ window: _ServiceHealthWindowApi+ /** Counts over all scored services. */+ summary: _ServiceHealthSummaryApi+ /** One row per service, in orderBy order. */+ services: _ServiceHealthRowApi[]+ /** Prefilled follow-up calls. */+ next: _ServiceHealthNextCallApi[]+ /** True when more services matched than limit. */+ truncated: boolean+ /** Caveats about this result. */+ notes: string[]+}+export interface _TracingSparklineQueryBodyApi {/** Date range for the query. Defaults to last hour. */dateRange?: _TracingDateRangeApi@@ -1248,6 +1504,21 @@ export type TracingSpansAttributesRetrieveParams = {* @minLength 1*/attribute_type?: TracingSpansAttributesRetrieveAttributeType+ /**+ * JSON-encoded date range object, e.g. {"date_from": "-1h"}. Malformed JSON returns 400. Or pass date_from and date_to.+ * @minLength 1+ */+ dateRange?: string+ /**+ * Range start: -1h, -7d or ISO 8601. Defaults to -1h.+ * @minLength 1+ */+ date_from?: string+ /**+ * Range end, same format. Omit for "now".+ * @minLength 1+ */+ date_to?: string/*** Max results (default: 100).* @minimum 1@@ -1280,10 +1551,20 @@ export const TracingSpansAttributesRetrieveAttributeType = {export type TracingSpansServiceNamesRetrieveParams = {/**- * JSON-encoded date range, e.g. '{"date_from": "-1h"}'.+ * JSON-encoded date range object, e.g. {"date_from": "-1h"}. Malformed JSON returns 400. Or pass date_from and date_to.* @minLength 1*/dateRange?: string+ /**+ * Range start: -1h, -7d or ISO 8601. Defaults to -1h.+ * @minLength 1+ */+ date_from?: string+ /**+ * Range end, same format. Omit for "now".+ * @minLength 1+ */+ date_to?: string/*** Search filter for service names.* @minLength 1@@ -1301,6 +1582,21 @@ export type TracingSpansValuesRetrieveParams = {* @minLength 1*/attribute_type?: TracingSpansValuesRetrieveAttributeType+ /**+ * JSON-encoded date range object, e.g. {"date_from": "-1h"}. Malformed JSON returns 400. Or pass date_from and date_to.+ * @minLength 1+ */+ dateRange?: string+ /**+ * Range start: -1h, -7d or ISO 8601. Defaults to -1h.+ * @minLength 1+ */+ date_from?: string+ /**+ * Range end, same format. Omit for "now".+ * @minLength 1+ */+ date_to?: string/*** The attribute key to get values for.* @minLength 1diff --git a/products/tracing/frontend/generated/api.ts b/products/tracing/frontend/generated/api.tsindex bb3b3372e4b6..ae4862a14c7c 100644--- a/products/tracing/frontend/generated/api.ts+++ b/products/tracing/frontend/generated/api.ts@@ -17,6 +17,8 @@ import type {PatchedTeamTracingConfigApi,PatchedTracesRetentionRuleApi,PatchedTracingViewApi,+ ServiceHealthRequestApi,+ ServiceHealthResponseApi,TeamTracingConfigApi,TracesRetentionRuleApi,TracingRetentionRulesListParams,@@ -539,6 +541,23 @@ export const tracingSpansServiceNamesRetrieve = async (})}+export const getTracingSpansServiceHealthCreateUrl = (projectId: string) => {+ return `/api/projects/${projectId}/tracing/spans/service_health/`+}++export const tracingSpansServiceHealthCreate = async (+ projectId: string,+ serviceHealthRequestApi: ServiceHealthRequestApi,+ options?: RequestInit+): Promise<ServiceHealthResponseApi> => {+ return apiMutator<ServiceHealthResponseApi>(getTracingSpansServiceHealthCreateUrl(projectId), {+ ...options,+ method: 'POST',+ headers: { 'Content-Type': 'application/json', ...options?.headers },+ body: JSON.stringify(serviceHealthRequestApi),+ })+}+export const getTracingSpansSparklineCreateUrl = (projectId: string) => {return `/api/projects/${projectId}/tracing/spans/sparkline/`}diff --git a/products/tracing/frontend/generated/api.zod.ts b/products/tracing/frontend/generated/api.zod.tsindex afa17ebe7542..1496c8b1913a 100644--- a/products/tracing/frontend/generated/api.zod.ts+++ b/products/tracing/frontend/generated/api.zod.ts@@ -1045,6 +1045,77 @@ export const TracingSpansQueryCreateBody = /* @__PURE__ */ zod.object({.describe('The tracing spans query to execute.'),})+export const tracingSpansServiceHealthCreateBodyQueryOneCompareDefault = `previous_period`+export const tracingSpansServiceHealthCreateBodyQueryOneServiceNamesMax = 100++export const tracingSpansServiceHealthCreateBodyQueryOneSpanScopeDefault = `entry`+export const tracingSpansServiceHealthCreateBodyQueryOneOrderByDefault = `status`+export const tracingSpansServiceHealthCreateBodyQueryOneLimitDefault = 25+export const tracingSpansServiceHealthCreateBodyQueryOneLimitMax = 200++export const TracingSpansServiceHealthCreateBody = /* @__PURE__ */ zod.object({+ query: zod+ .object({+ dateRange: zod+ .object({+ date_from: zod.string().nullish().describe('Relative (-30m, -6h, -7d) or ISO 8601.'),+ date_to: zod.string().nullish().describe('Same format. Omit or null for \"now\".'),+ })+ .optional()+ .describe('The window to score. Defaults to the last hour.'),+ compare: zod+ .enum(['none', 'previous_period', 'previous_day', 'previous_week'])+ .describe(+ '\* `none` - None\n\* `previous_period` - Previous Period\n\* `previous_day` - Previous Day\n\* `previous_week` - Previous Week'+ )+ .default(tracingSpansServiceHealthCreateBodyQueryOneCompareDefault)+ .describe(+ 'The baseline. previous_period (default) is the same length just before the window. previous_day and previous_week shift the window back 1 or 7 days. none skips it. baselineDateRange overrides it.\n\n\* `none` - None\n\* `previous_period` - Previous Period\n\* `previous_day` - Previous Day\n\* `previous_week` - Previous Week'+ ),+ environment: zod+ .string()+ .nullish()+ .describe(+ 'Only this environment, matched without case on deployment.environment.name, then deployment.environment, then env.'+ ),+ serviceNames: zod+ .array(zod.string())+ .max(tracingSpansServiceHealthCreateBodyQueryOneServiceNamesMax)+ .optional()+ .describe('Only these services.'),+ spanScope: zod+ .enum(['entry', 'all'])+ .describe('\* `entry` - Entry\n\* `all` - All')+ .default(tracingSpansServiceHealthCreateBodyQueryOneSpanScopeDefault)+ .describe(+ 'entry (default) counts root, server and consumer spans, one per request. all counts every span.\n\n\* `entry` - Entry\n\* `all` - All'+ ),+ orderBy: zod+ .enum(['status', 'error_rate', 'p95', 'throughput', 'error_count'])+ .describe(+ '\* `status` - Status\n\* `error_rate` - Error Rate\n\* `p95` - P95\n\* `throughput` - Throughput\n\* `error_count` - Error Count'+ )+ .default(tracingSpansServiceHealthCreateBodyQueryOneOrderByDefault)+ .describe(+ 'status (default) puts silent, regressed and degraded first. The others sort high to low.\n\n\* `status` - Status\n\* `error_rate` - Error Rate\n\* `p95` - P95\n\* `throughput` - Throughput\n\* `error_count` - Error Count'+ ),+ limit: zod+ .number()+ .min(1)+ .max(tracingSpansServiceHealthCreateBodyQueryOneLimitMax)+ .default(tracingSpansServiceHealthCreateBodyQueryOneLimitDefault)+ .describe('Max service rows, 1-200.'),+ })+ .describe('The service health query. Every field is optional.'),+ baselineDateRange: zod+ .object({+ date_from: zod.string().nullish().describe('Relative (-30m, -6h, -7d) or ISO 8601.'),+ date_to: zod.string().nullish().describe('Same format. Omit or null for \"now\".'),+ })+ .optional()+ .describe('An explicit baseline window. Overrides query.compare.'),+})+export const tracingSpansSparklineCreateBodyQueryOneFilterGroupDefault = []export const tracingSpansSparklineCreateBodyQueryOneRootSpansDefault = falsediff --git a/products/tracing/mcp/prompts/apm-service-health.md b/products/tracing/mcp/prompts/apm-service-health.mdnew file mode 100644index 000000000000..21b93201da23--- /dev/null+++ b/products/tracing/mcp/prompts/apm-service-health.md@@ -0,0 +1,18 @@+Score every service in one call: requests, throughput, error rate, p50/p95/p99 and a status, compared with a baseline window.++Use this first for "how healthy are my services" or a daily report.+Use `apm-spans-aggregate` next for the operations inside one service. Use `apm-spans-count` only for a raw count.++All parameters go inside `query`, and every one is optional:++```json+{ "query": { "dateRange": { "date_from": "-24h" }, "compare": "previous_day", "environment": "production" } }+```++Each row has a `status`: `silent` (spans stopped), `regressed` (error rate or p95 got worse), `degraded` (error rate 5% or more), `new`, `recovered`, `improved` or `ok`, and a short `reason`.+Rows come worst first.++`window` gives the absolute UTC times the server used. Reuse `window.current` as `dateRange` in follow-up calls.+`next` holds prefilled follow-up calls for regressed and degraded services. It is empty when `environment` is set, because those tools cannot repeat the environment fallback.+`spanScope: "entry"` (default) counts one span per request. Use `"all"` to count every span.+A 400 that names the 10 GB cap means the window is too wide. Use the window it suggests, or add `serviceNames`.diff --git a/products/tracing/mcp/prompts/apm-spans-count.md b/products/tracing/mcp/prompts/apm-spans-count.mdindex 809065db32c4..db352201b39d 100644--- a/products/tracing/mcp/prompts/apm-spans-count.md+++ b/products/tracing/mcp/prompts/apm-spans-count.md@@ -12,7 +12,7 @@ All parameters go inside `query` — top-level fields are rejected:- When the user asks "how many X spans are there?" and you don't need to see individual spans.- To check whether a filter combination matches anything at all before committing to a full query.-This counts **spans**, not traces. A single trace contains many spans, so a count of matching spans will exceed the number of matching traces.+`count` is the number of matching **spans**. `traceCount` is the number of distinct traces whose root span matches. A trace contains many spans, so `count` is usually larger.If the count would scan too much data (a wide date range with no filters), the tool returns a 400 asking you to narrow the window or add filters — narrow the `dateRange` or add `serviceNames` / `statusCodes` / `filterGroup`, then retry.diff --git a/products/tracing/mcp/tools.yaml b/products/tracing/mcp/tools.yamlindex af54ba930202..f19af110ea47 100644--- a/products/tracing/mcp/tools.yaml+++ b/products/tracing/mcp/tools.yaml@@ -68,6 +68,18 @@ tools:include:- results- count+ apm-service-health:+ operation: tracing_spans_service_health_create+ enabled: true+ scopes:+ - tracing:read+ annotations:+ readOnly: true+ destructive: false+ idempotent: true+ title: Service health against a baseline+ description_file: ./prompts/apm-service-health.md+ feature_flag: tracingapm-services-list:operation: tracing_spans_service_names_retrieveenabled: true@@ -121,6 +133,7 @@ tools:response:include:- count+ - traceCountapm-spans-duration-histogram:operation: tracing_spans_duration_histogram_createenabled: truediff --git a/products/tracing/skills/exploring-apm-traces/SKILL.md b/products/tracing/skills/exploring-apm-traces/SKILL.mdindex 404c3df993b2..6e3967d21b36 100644--- a/products/tracing/skills/exploring-apm-traces/SKILL.md+++ b/products/tracing/skills/exploring-apm-traces/SKILL.md@@ -4,7 +4,7 @@ description: >Investigates distributed application performance using PostHog APM (OpenTelemetry span) data via MCP.Use when the user asks about service traces, slow HTTP/database spans, error spans, error-rate trends orspikes, latency distributions, trace IDs, or span attributes — not AI observability traces or product logs.- Uses posthog:query-apm-spans, posthog:apm-trace-get, posthog:apm-spans-sparkline,+ Uses posthog:apm-service-health, posthog:query-apm-spans, posthog:apm-trace-get, posthog:apm-spans-sparkline,posthog:apm-services-list, posthog:apm-attributes-list, and posthog:apm-attribute-values-list.---@@ -22,6 +22,7 @@ When the question asks for SLO burn, call `posthog:metric-list` before the APM t| Tool | Purpose || -------------------------------------- | ------------------------------------------------- |+| `posthog:apm-service-health` | Status per service against a baseline: start here || `posthog:query-apm-spans` | Search and filter spans (compact list view) || `posthog:apm-trace-get` | Get the full span list for one hex `trace_id` || `posthog:apm-spans-aggregate` | Per-operation aggregates (count, p50/p95, errors) |@@ -99,6 +100,19 @@ To rebuild the tree:## Investigation patterns+### "How healthy are my services?" (start here)++1. `apm-service-health` scores every service in one call: requests, error rate, p50/p95/p99, the change from a baseline window, and a `status` (`silent`, `regressed`, `degraded`, `new`, `recovered`, `improved`, `ok`) with a `reason`. Rows come worst first.+2. Pick the baseline with `compare`: `previous_period` (default), `previous_day`, `previous_week`, or an explicit `baselineDateRange` next to `query`. Scope with `environment` and `serviceNames`.+3. Run the calls in `next`. They are prefilled: `apm-spans-aggregate` for the operations behind the regressed and degraded services, `apm-spans-sparkline` for when a silent service stopped, `query-apm-spans` for error spans.+4. Do not build a service summary from `apm-spans-aggregate` rows or from one `apm-spans-count` call per service. Percentiles of operation rows do not add up to a service percentile.++#### Daily service health report++1. `apm-service-health` with `dateRange: {"date_from": "-24h"}`, and `compare: "previous_day"`. Follow its `next` calls to the operations behind each unhealthy service.+2. `logs-patterns-diff` for new error patterns in the services that are not `ok`.+3. `logs-alerts-list` for alerts that fired in the window.+### "Where is time going?"1. Every span from `apm-trace-get` carries `self_time_nano` — duration not covered by children. Sort by it: the top span is where wall-clock actually went. A parent with large `self_time_nano` is an **uninstrumented gap** (the work happened inside it, not in any recorded child).@@ -116,7 +130,7 @@ To rebuild the tree:### "Did the request hit service X?"1. Run `print_summary.py` — it prints the set of services involved in the trace.-2. If service X is missing, the request never reached it (or instrumentation is missing — check `apm-services-list` to confirm X has emitted spans recently at all).+2. If service X is missing, the request never reached it (or instrumentation is missing — check `apm-service-health` with `serviceNames: ["X"]` to see whether X emitted spans recently, and when it stopped).### "What's different about the bad spans?" (over-represented values)diff --git a/services/mcp/schema/generated-tool-definitions.json b/services/mcp/schema/generated-tool-definitions.jsonindex 2cb99fa190f0..9cf118a19357 100644--- a/services/mcp/schema/generated-tool-definitions.json+++ b/services/mcp/schema/generated-tool-definitions.json@@ -941,6 +941,21 @@},"feature_flag": "tracing"},+ "apm-service-health": {+ "description": "Score every service in one call: requests, throughput, error rate, p50/p95/p99 and a status, compared with a baseline window.\n\nUse this first for \"how healthy are my services\" or a daily report.\nUse `apm-spans-aggregate` next for the operations inside one service. Use `apm-spans-count` only for a raw count.\n\nAll parameters go inside `query`, and every one is optional:\n\n```json\n{ \"query\": { \"dateRange\": { \"date_from\": \"-24h\" }, \"compare\": \"previous_day\", \"environment\": \"production\" } }\n```\n\nEach row has a `status`: `silent` (spans stopped), `regressed` (error rate or p95 got worse), `degraded` (error rate 5% or more), `new`, `recovered`, `improved` or `ok`, and a short `reason`.\nRows come worst first.\n\n`window` gives the absolute UTC times the server used. Reuse `window.current` as `dateRange` in follow-up calls.\n`next` holds prefilled follow-up calls for regressed and degraded services. It is empty when `environment` is set, because those tools cannot repeat the environment fallback.\n`spanScope: \"entry\"` (default) counts one span per request. Use `\"all\"` to count every span.\nA 400 that names the 10 GB cap means the window is too wide. Use the window it suggests, or add `serviceNames`.",+ "category": "Tracing",+ "feature": "tracing",+ "summary": "Service health against a baseline",+ "title": "Service health against a baseline",+ "required_scopes": ["tracing:read"],+ "annotations": {+ "destructiveHint": false,+ "idempotentHint": true,+ "openWorldHint": true,+ "readOnlyHint": true+ },+ "feature_flag": "tracing"+ },"apm-services-list": {"description": "List distinct service names that have emitted trace spans. Use to discover available services before filtering spans by service name.","category": "Tracing",@@ -972,7 +987,7 @@"feature_flag": "tracing"},"apm-spans-count": {- "description": "Return a scalar count of trace spans matching a filter set. Use this as a cheap pre-flight before `query-apm-spans` — if the count is large, narrow the filters (or set `excludeAttributes`) before pulling rows.\n\nAll parameters go inside `query` — top-level fields are rejected:\n\n```json\n{ \"query\": { \"serviceNames\": [\"api\"], \"dateRange\": { \"date_from\": \"-1h\" } } }\n```\n\n# When to use\n\n- Before `query-apm-spans`, to confirm the filter set returns a tractable number of spans rather than pulling rows blind.\n- When the user asks \"how many X spans are there?\" and you don't need to see individual spans.\n- To check whether a filter combination matches anything at all before committing to a full query.\n\nThis counts **spans**, not traces. A single trace contains many spans, so a count of matching spans will exceed the number of matching traces.\n\nIf the count would scan too much data (a wide date range with no filters), the tool returns a 400 asking you to narrow the window or add filters — narrow the `dateRange` or add `serviceNames` / `statusCodes` / `filterGroup`, then retry.\n\n# Parameters\n\n## query.dateRange\n\nDate range for the count. Defaults to the last hour (`-1h`).\n\n- `date_from`: Start of the range. Accepts ISO 8601 timestamps or relative formats: `-1h`, `-6h`, `-1d`, `-7d`, `-30d`.\n- `date_to`: End of the range. Same format. Omit or null for \"now\".\n\n## query.serviceNames\n\nFilter by service names. Unlike `query-apm-spans`, an unfiltered count is cheap and useful for sizing up a filter before narrowing. Use `apm-services-list` to discover services.\n\n## query.statusCodes\n\nFilter by OTel span status codes (list of integers: `0` Unset, `1` OK, `2` Error) — **not** HTTP status codes. Use `[2]` to select error spans.\n\n## query.filterGroup\n\nProperty filters to narrow the count. Same format as `query-apm-spans` filters — each filter specifies `key`, `operator`, `type` (span/span_attribute/span_resource_attribute), and optionally `value`.\n\nA `span` filter on `duration` takes a value in milliseconds. For example, `{ \"key\": \"duration\", \"operator\": \"gt\", \"type\": \"span\", \"value\": \"30000\" }` counts spans longer than 30 seconds.\n\n# Examples\n\n## Count error spans in a service over the last day\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"statusCodes\": [2],\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\n## Count how many spans match a name before fetching them\n\n```json\n{\n \"query\": {\n \"filterGroup\": [{ \"key\": \"name\", \"operator\": \"exact\", \"type\": \"span\", \"value\": \"redis_cluster.discovery\" }],\n \"dateRange\": { \"date_from\": \"-6h\" }\n }\n}\n```",+ "description": "Return a scalar count of trace spans matching a filter set. Use this as a cheap pre-flight before `query-apm-spans` — if the count is large, narrow the filters (or set `excludeAttributes`) before pulling rows.\n\nAll parameters go inside `query` — top-level fields are rejected:\n\n```json\n{ \"query\": { \"serviceNames\": [\"api\"], \"dateRange\": { \"date_from\": \"-1h\" } } }\n```\n\n# When to use\n\n- Before `query-apm-spans`, to confirm the filter set returns a tractable number of spans rather than pulling rows blind.\n- When the user asks \"how many X spans are there?\" and you don't need to see individual spans.\n- To check whether a filter combination matches anything at all before committing to a full query.\n\n`count` is the number of matching **spans**. `traceCount` is the number of distinct traces whose root span matches. A trace contains many spans, so `count` is usually larger.\n\nIf the count would scan too much data (a wide date range with no filters), the tool returns a 400 asking you to narrow the window or add filters — narrow the `dateRange` or add `serviceNames` / `statusCodes` / `filterGroup`, then retry.\n\n# Parameters\n\n## query.dateRange\n\nDate range for the count. Defaults to the last hour (`-1h`).\n\n- `date_from`: Start of the range. Accepts ISO 8601 timestamps or relative formats: `-1h`, `-6h`, `-1d`, `-7d`, `-30d`.\n- `date_to`: End of the range. Same format. Omit or null for \"now\".\n\n## query.serviceNames\n\nFilter by service names. Unlike `query-apm-spans`, an unfiltered count is cheap and useful for sizing up a filter before narrowing. Use `apm-services-list` to discover services.\n\n## query.statusCodes\n\nFilter by OTel span status codes (list of integers: `0` Unset, `1` OK, `2` Error) — **not** HTTP status codes. Use `[2]` to select error spans.\n\n## query.filterGroup\n\nProperty filters to narrow the count. Same format as `query-apm-spans` filters — each filter specifies `key`, `operator`, `type` (span/span_attribute/span_resource_attribute), and optionally `value`.\n\nA `span` filter on `duration` takes a value in milliseconds. For example, `{ \"key\": \"duration\", \"operator\": \"gt\", \"type\": \"span\", \"value\": \"30000\" }` counts spans longer than 30 seconds.\n\n# Examples\n\n## Count error spans in a service over the last day\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"statusCodes\": [2],\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\n## Count how many spans match a name before fetching them\n\n```json\n{\n \"query\": {\n \"filterGroup\": [{ \"key\": \"name\", \"operator\": \"exact\", \"type\": \"span\", \"value\": \"redis_cluster.discovery\" }],\n \"dateRange\": { \"date_from\": \"-6h\" }\n }\n}\n```","category": "Tracing","feature": "tracing","summary": "Count spans",diff --git a/services/mcp/schema/tool-definitions-all.json b/services/mcp/schema/tool-definitions-all.jsonindex dd7c19b0b216..06d4a6fa02b4 100644--- a/services/mcp/schema/tool-definitions-all.json+++ b/services/mcp/schema/tool-definitions-all.json@@ -956,6 +956,21 @@},"feature_flag": "tracing"},+ "apm-service-health": {+ "description": "Score every service in one call: requests, throughput, error rate, p50/p95/p99 and a status, compared with a baseline window.\n\nUse this first for \"how healthy are my services\" or a daily report.\nUse `apm-spans-aggregate` next for the operations inside one service. Use `apm-spans-count` only for a raw count.\n\nAll parameters go inside `query`, and every one is optional:\n\n```json\n{ \"query\": { \"dateRange\": { \"date_from\": \"-24h\" }, \"compare\": \"previous_day\", \"environment\": \"production\" } }\n```\n\nEach row has a `status`: `silent` (spans stopped), `regressed` (error rate or p95 got worse), `degraded` (error rate 5% or more), `new`, `recovered`, `improved` or `ok`, and a short `reason`.\nRows come worst first.\n\n`window` gives the absolute UTC times the server used. Reuse `window.current` as `dateRange` in follow-up calls.\n`next` holds prefilled follow-up calls for regressed and degraded services. It is empty when `environment` is set, because those tools cannot repeat the environment fallback.\n`spanScope: \"entry\"` (default) counts one span per request. Use `\"all\"` to count every span.\nA 400 that names the 10 GB cap means the window is too wide. Use the window it suggests, or add `serviceNames`.",+ "category": "Tracing",+ "feature": "tracing",+ "summary": "Service health against a baseline",+ "title": "Service health against a baseline",+ "required_scopes": ["tracing:read"],+ "annotations": {+ "destructiveHint": false,+ "idempotentHint": true,+ "openWorldHint": true,+ "readOnlyHint": true+ },+ "feature_flag": "tracing"+ },"apm-services-list": {"description": "List distinct service names that have emitted trace spans. Use to discover available services before filtering spans by service name.","category": "Tracing",@@ -987,7 +1002,7 @@"feature_flag": "tracing"},"apm-spans-count": {- "description": "Return a scalar count of trace spans matching a filter set. Use this as a cheap pre-flight before `query-apm-spans` — if the count is large, narrow the filters (or set `excludeAttributes`) before pulling rows.\n\nAll parameters go inside `query` — top-level fields are rejected:\n\n```json\n{ \"query\": { \"serviceNames\": [\"api\"], \"dateRange\": { \"date_from\": \"-1h\" } } }\n```\n\n# When to use\n\n- Before `query-apm-spans`, to confirm the filter set returns a tractable number of spans rather than pulling rows blind.\n- When the user asks \"how many X spans are there?\" and you don't need to see individual spans.\n- To check whether a filter combination matches anything at all before committing to a full query.\n\nThis counts **spans**, not traces. A single trace contains many spans, so a count of matching spans will exceed the number of matching traces.\n\nIf the count would scan too much data (a wide date range with no filters), the tool returns a 400 asking you to narrow the window or add filters — narrow the `dateRange` or add `serviceNames` / `statusCodes` / `filterGroup`, then retry.\n\n# Parameters\n\n## query.dateRange\n\nDate range for the count. Defaults to the last hour (`-1h`).\n\n- `date_from`: Start of the range. Accepts ISO 8601 timestamps or relative formats: `-1h`, `-6h`, `-1d`, `-7d`, `-30d`.\n- `date_to`: End of the range. Same format. Omit or null for \"now\".\n\n## query.serviceNames\n\nFilter by service names. Unlike `query-apm-spans`, an unfiltered count is cheap and useful for sizing up a filter before narrowing. Use `apm-services-list` to discover services.\n\n## query.statusCodes\n\nFilter by OTel span status codes (list of integers: `0` Unset, `1` OK, `2` Error) — **not** HTTP status codes. Use `[2]` to select error spans.\n\n## query.filterGroup\n\nProperty filters to narrow the count. Same format as `query-apm-spans` filters — each filter specifies `key`, `operator`, `type` (span/span_attribute/span_resource_attribute), and optionally `value`.\n\nA `span` filter on `duration` takes a value in milliseconds. For example, `{ \"key\": \"duration\", \"operator\": \"gt\", \"type\": \"span\", \"value\": \"30000\" }` counts spans longer than 30 seconds.\n\n# Examples\n\n## Count error spans in a service over the last day\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"statusCodes\": [2],\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\n## Count how many spans match a name before fetching them\n\n```json\n{\n \"query\": {\n \"filterGroup\": [{ \"key\": \"name\", \"operator\": \"exact\", \"type\": \"span\", \"value\": \"redis_cluster.discovery\" }],\n \"dateRange\": { \"date_from\": \"-6h\" }\n }\n}\n```",+ "description": "Return a scalar count of trace spans matching a filter set. Use this as a cheap pre-flight before `query-apm-spans` — if the count is large, narrow the filters (or set `excludeAttributes`) before pulling rows.\n\nAll parameters go inside `query` — top-level fields are rejected:\n\n```json\n{ \"query\": { \"serviceNames\": [\"api\"], \"dateRange\": { \"date_from\": \"-1h\" } } }\n```\n\n# When to use\n\n- Before `query-apm-spans`, to confirm the filter set returns a tractable number of spans rather than pulling rows blind.\n- When the user asks \"how many X spans are there?\" and you don't need to see individual spans.\n- To check whether a filter combination matches anything at all before committing to a full query.\n\n`count` is the number of matching **spans**. `traceCount` is the number of distinct traces whose root span matches. A trace contains many spans, so `count` is usually larger.\n\nIf the count would scan too much data (a wide date range with no filters), the tool returns a 400 asking you to narrow the window or add filters — narrow the `dateRange` or add `serviceNames` / `statusCodes` / `filterGroup`, then retry.\n\n# Parameters\n\n## query.dateRange\n\nDate range for the count. Defaults to the last hour (`-1h`).\n\n- `date_from`: Start of the range. Accepts ISO 8601 timestamps or relative formats: `-1h`, `-6h`, `-1d`, `-7d`, `-30d`.\n- `date_to`: End of the range. Same format. Omit or null for \"now\".\n\n## query.serviceNames\n\nFilter by service names. Unlike `query-apm-spans`, an unfiltered count is cheap and useful for sizing up a filter before narrowing. Use `apm-services-list` to discover services.\n\n## query.statusCodes\n\nFilter by OTel span status codes (list of integers: `0` Unset, `1` OK, `2` Error) — **not** HTTP status codes. Use `[2]` to select error spans.\n\n## query.filterGroup\n\nProperty filters to narrow the count. Same format as `query-apm-spans` filters — each filter specifies `key`, `operator`, `type` (span/span_attribute/span_resource_attribute), and optionally `value`.\n\nA `span` filter on `duration` takes a value in milliseconds. For example, `{ \"key\": \"duration\", \"operator\": \"gt\", \"type\": \"span\", \"value\": \"30000\" }` counts spans longer than 30 seconds.\n\n# Examples\n\n## Count error spans in a service over the last day\n\n```json\n{\n \"query\": {\n \"serviceNames\": [\"api-gateway\"],\n \"statusCodes\": [2],\n \"dateRange\": { \"date_from\": \"-1d\" }\n }\n}\n```\n\n## Count how many spans match a name before fetching them\n\n```json\n{\n \"query\": {\n \"filterGroup\": [{ \"key\": \"name\", \"operator\": \"exact\", \"type\": \"span\", \"value\": \"redis_cluster.discovery\" }],\n \"dateRange\": { \"date_from\": \"-6h\" }\n }\n}\n```","category": "Tracing","feature": "tracing","summary": "Count spans",diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.tsindex 234f0e1ae241..944cd64beb2a 100644--- a/services/mcp/src/api/generated.ts+++ b/services/mcp/src/api/generated.ts@@ -102886,6 +102886,264 @@ export namespace Schemas {policies?: ToolPolicyEntry[];}+ /**+ * * `none` - None+ * * `previous_period` - Previous Period+ * * `previous_day` - Previous Day+ * * `previous_week` - Previous Week+ */+ export type ServiceHealthBaselineEnum = typeof ServiceHealthBaselineEnum[keyof typeof ServiceHealthBaselineEnum];+++ export const ServiceHealthBaselineEnum = {+ None: 'none',+ PreviousPeriod: 'previous_period',+ PreviousDay: 'previous_day',+ PreviousWeek: 'previous_week',+ } as const;++ /**+ * * `status` - Status+ * * `error_rate` - Error Rate+ * * `p95` - P95+ * * `throughput` - Throughput+ * * `error_count` - Error Count+ */+ export type ServiceHealthOrderByEnum = typeof ServiceHealthOrderByEnum[keyof typeof ServiceHealthOrderByEnum];+++ export const ServiceHealthOrderByEnum = {+ Status: 'status',+ ErrorRate: 'error_rate',+ P95: 'p95',+ Throughput: 'throughput',+ ErrorCount: 'error_count',+ } as const;++ export interface _ServiceHealthDateRange {+ /**+ * Relative (-30m, -6h, -7d) or ISO 8601.+ * @nullable+ */+ date_from?: string | null;+ /**+ * Same format. Omit or null for "now".+ * @nullable+ */+ date_to?: string | null;+ }++ /**+ * * `entry` - Entry+ * * `all` - All+ */+ export type ServiceHealthSpanScopeEnum = typeof ServiceHealthSpanScopeEnum[keyof typeof ServiceHealthSpanScopeEnum];+++ export const ServiceHealthSpanScopeEnum = {+ Entry: 'entry',+ All: 'all',+ } as const;++ export interface _ServiceHealthQuery {+ /** The window to score. Defaults to the last hour. */+ dateRange?: _ServiceHealthDateRange;+ /** The baseline. previous_period (default) is the same length just before the window. previous_day and previous_week shift the window back 1 or 7 days. none skips it. baselineDateRange overrides it.+ *+ * * `none` - None+ * * `previous_period` - Previous Period+ * * `previous_day` - Previous Day+ * * `previous_week` - Previous Week */+ compare?: ServiceHealthBaselineEnum;+ /**+ * Only this environment, matched without case on deployment.environment.name, then deployment.environment, then env.+ * @nullable+ */+ environment?: string | null;+ /**+ * Only these services.+ * @maxItems 100+ */+ serviceNames?: string[];+ /** entry (default) counts root, server and consumer spans, one per request. all counts every span.+ *+ * * `entry` - Entry+ * * `all` - All */+ spanScope?: ServiceHealthSpanScopeEnum;+ /** status (default) puts silent, regressed and degraded first. The others sort high to low.+ *+ * * `status` - Status+ * * `error_rate` - Error Rate+ * * `p95` - P95+ * * `throughput` - Throughput+ * * `error_count` - Error Count */+ orderBy?: ServiceHealthOrderByEnum;+ /**+ * Max service rows, 1-200.+ * @minimum 1+ * @maximum 200+ */+ limit?: number;+ }++ export interface ServiceHealthRequest {+ /** The service health query. Every field is optional. */+ query: _ServiceHealthQuery;+ /** An explicit baseline window. Overrides query.compare. */+ baselineDateRange?: _ServiceHealthDateRange;+ }++ export interface _ServiceHealthTimeRange {+ /** Start, ISO 8601 UTC, inclusive. */+ date_from: string;+ /** End, ISO 8601 UTC, exclusive. */+ date_to: string;+ }++ export interface _ServiceHealthWindow {+ /** The scored window. Reuse it as dateRange in follow-up calls. */+ current: _ServiceHealthTimeRange;+ /** The baseline window. Null when compare is none. */+ baseline: _ServiceHealthTimeRange | null;+ /** Always UTC. */+ timezone: string;+ }++ export interface _ServiceHealthSummary {+ /** Services scored, before the limit. */+ services: number;+ /** Requests over all scored services. */+ requests: number;+ /** Services per status. */+ ok: number;+ /** Services per status. */+ degraded: number;+ /** Services per status. */+ regressed: number;+ /** Services per status. */+ silent: number;+ /** Services per status. */+ new: number;+ /** Services per status. */+ recovered: number;+ /** Services per status. */+ improved: number;+ }++ /**+ * * `silent` - Silent+ * * `regressed` - Regressed+ * * `degraded` - Degraded+ * * `new` - New+ * * `recovered` - Recovered+ * * `improved` - Improved+ * * `ok` - Ok+ */+ export type ServiceHealthStatusEnum = typeof ServiceHealthStatusEnum[keyof typeof ServiceHealthStatusEnum];+++ export const ServiceHealthStatusEnum = {+ Silent: 'silent',+ Regressed: 'regressed',+ Degraded: 'degraded',+ New: 'new',+ Recovered: 'recovered',+ Improved: 'improved',+ Ok: 'ok',+ } as const;++ export interface _ServiceHealthRow {+ /** Service name. */+ service: string;+ /** silent: spans stopped. regressed: error rate or p95 worse than the baseline. degraded: error rate 5% or more. new: no spans in the baseline. recovered, improved: the reverse of regressed.+ *+ * * `silent` - Silent+ * * `regressed` - Regressed+ * * `degraded` - Degraded+ * * `new` - New+ * * `recovered` - Recovered+ * * `improved` - Improved+ * * `ok` - Ok */+ status: ServiceHealthStatusEnum;+ /** Why the service has this status. Empty for ok. */+ reason: string;+ /** Spans in the window, in the chosen spanScope. */+ requests: number;+ /** Requests per minute. */+ throughput_per_min: number;+ /** Requests with OTel status Error. */+ error_count: number;+ /**+ * Error share of requests, percent.+ * @nullable+ */+ error_rate_pct: number | null;+ /**+ * Change from the baseline, points.+ * @nullable+ */+ error_rate_delta_pp: number | null;+ /**+ * Median duration, ms.+ * @nullable+ */+ p50_ms: number | null;+ /**+ * 95th percentile duration, ms.+ * @nullable+ */+ p95_ms: number | null;+ /**+ * p95 change from the baseline, percent.+ * @nullable+ */+ p95_change_pct: number | null;+ /**+ * 99th percentile duration, ms.+ * @nullable+ */+ p99_ms: number | null;+ /**+ * Baseline requests. Null without one.+ * @nullable+ */+ baseline_requests: number | null;+ /**+ * Last span, ISO 8601 UTC.+ * @nullable+ */+ last_seen: string | null;+ }++ /**+ * The prefilled query argument.+ */+ export type _ServiceHealthNextCallQuery = { [key: string]: unknown };++ export interface _ServiceHealthNextCall {+ /** The tool to call next. */+ tool: string;+ /** What the call answers. */+ why: string;+ /** The prefilled query argument. */+ query: _ServiceHealthNextCallQuery;+ }++ export interface ServiceHealthResponse {+ /** The absolute UTC windows the server used. */+ window: _ServiceHealthWindow;+ /** Counts over all scored services. */+ summary: _ServiceHealthSummary;+ /** One row per service, in orderBy order. */+ services: _ServiceHealthRow[];+ /** Prefilled follow-up calls. */+ next: _ServiceHealthNextCall[];+ /** True when more services matched than limit. */+ truncated: boolean;+ /** Caveats about this result. */+ notes: string[];+ }+export interface SessionRecordingBulkDeleteRequest {/*** Session IDs of the recordings to delete (max 100 per call).@@ -133384,6 +133642,21 @@ export namespace Schemas {* @minLength 1*/attribute_type?: TracingSpansAttributesRetrieveAttributeType;+ /**+ * JSON-encoded date range object, e.g. {"date_from": "-1h"}. Malformed JSON returns 400. Or pass date_from and date_to.+ * @minLength 1+ */+ dateRange?: string;+ /**+ * Range start: -1h, -7d or ISO 8601. Defaults to -1h.+ * @minLength 1+ */+ date_from?: string;+ /**+ * Range end, same format. Omit for "now".+ * @minLength 1+ */+ date_to?: string;/*** Max results (default: 100).* @minimum 1@@ -133416,10 +133689,20 @@ export namespace Schemas {export type TracingSpansServiceNamesRetrieveParams = {/**- * JSON-encoded date range, e.g. '{"date_from": "-1h"}'.+ * JSON-encoded date range object, e.g. {"date_from": "-1h"}. Malformed JSON returns 400. Or pass date_from and date_to.* @minLength 1*/dateRange?: string;+ /**+ * Range start: -1h, -7d or ISO 8601. Defaults to -1h.+ * @minLength 1+ */+ date_from?: string;+ /**+ * Range end, same format. Omit for "now".+ * @minLength 1+ */+ date_to?: string;/*** Search filter for service names.* @minLength 1@@ -133437,6 +133720,21 @@ export namespace Schemas {* @minLength 1*/attribute_type?: TracingSpansValuesRetrieveAttributeType;+ /**+ * JSON-encoded date range object, e.g. {"date_from": "-1h"}. Malformed JSON returns 400. Or pass date_from and date_to.+ * @minLength 1+ */+ dateRange?: string;+ /**+ * Range start: -1h, -7d or ISO 8601. Defaults to -1h.+ * @minLength 1+ */+ date_from?: string;+ /**+ * Range end, same format. Omit for "now".+ * @minLength 1+ */+ date_to?: string;/*** The attribute key to get values for.* @minLength 1diff --git a/services/mcp/src/generated/tracing/api.ts b/services/mcp/src/generated/tracing/api.tsindex db1403209a1f..65616b6f3475 100644--- a/services/mcp/src/generated/tracing/api.ts+++ b/services/mcp/src/generated/tracing/api.ts@@ -3,7 +3,7 @@* MCP service uses these Zod schemas for generated tool handlers.* To regenerate: hogli build:openapi*- * PostHog API - MCP 12 enabled ops+ * PostHog API - MCP 13 enabled ops* OpenAPI spec version: 1.0.0*/import * as zod from 'zod'@@ -292,6 +292,15 @@ export const TracingSpansAttributesRetrieveQueryParams = () => zod.object({.describe('Type of attributes: \"span_attribute\" for span-level attributes, \"span_resource_attribute\" for resource-level attributes.\n\n\* `span_attribute` - span_attribute\n\* `span_resource_attribute` - span_resource_attribute'),+ dateRange: zod+ .string()+ .min(1)+ .optional()+ .describe(+ 'JSON-encoded date range object, e.g. {\"date_from\": \"-1h\"}. Malformed JSON returns 400. Or pass date_from and date_to.'+ ),+ date_from: zod.string().min(1).optional().describe('Range start: -1h, -7d or ISO 8601. Defaults to -1h.'),+ date_to: zod.string().min(1).optional().describe('Range end, same format. Omit for \"now\".'),limit: zod.number().min(1)@@ -802,10 +811,97 @@ export const TracingSpansServiceNamesRetrieveParams = () => zod.object({})export const TracingSpansServiceNamesRetrieveQueryParams = () => zod.object({- dateRange: zod.string().min(1).optional().describe('JSON-encoded date range, e.g. \'{\"date_from\": \"-1h\"}\'.'),+ dateRange: zod+ .string()+ .min(1)+ .optional()+ .describe(+ 'JSON-encoded date range object, e.g. {\"date_from\": \"-1h\"}. Malformed JSON returns 400. Or pass date_from and date_to.'+ ),+ date_from: zod.string().min(1).optional().describe('Range start: -1h, -7d or ISO 8601. Defaults to -1h.'),+ date_to: zod.string().min(1).optional().describe('Range end, same format. Omit for \"now\".'),search: zod.string().min(1).optional().describe('Search filter for service names.'),})+export const TracingSpansServiceHealthCreateParams = () => zod.object({+ project_id: zod+ .string()+ .describe(+ "Project ID of the project you're trying to access. To find the ID of the project, make a call to \/api\/projects\/."+ ),+})++export const tracingSpansServiceHealthCreateBodyQueryOneCompareDefault = `previous_period`+export const tracingSpansServiceHealthCreateBodyQueryOneServiceNamesMax = 100++export const tracingSpansServiceHealthCreateBodyQueryOneSpanScopeDefault = `entry`+export const tracingSpansServiceHealthCreateBodyQueryOneOrderByDefault = `status`+export const tracingSpansServiceHealthCreateBodyQueryOneLimitDefault = 25+export const tracingSpansServiceHealthCreateBodyQueryOneLimitMax = 200++export const TracingSpansServiceHealthCreateBody = () => zod.object({+ query: zod+ .object({+ dateRange: zod+ .object({+ date_from: zod.string().nullish().describe('Relative (-30m, -6h, -7d) or ISO 8601.'),+ date_to: zod.string().nullish().describe('Same format. Omit or null for \"now\".'),+ })+ .optional()+ .describe('The window to score. Defaults to the last hour.'),+ compare: zod+ .enum(['none', 'previous_period', 'previous_day', 'previous_week'])+ .describe(+ '\* `none` - None\n\* `previous_period` - Previous Period\n\* `previous_day` - Previous Day\n\* `previous_week` - Previous Week'+ )+ .default(tracingSpansServiceHealthCreateBodyQueryOneCompareDefault)+ .describe(+ 'The baseline. previous_period (default) is the same length just before the window. previous_day and previous_week shift the window back 1 or 7 days. none skips it. baselineDateRange overrides it.\n\n\* `none` - None\n\* `previous_period` - Previous Period\n\* `previous_day` - Previous Day\n\* `previous_week` - Previous Week'+ ),+ environment: zod+ .string()+ .nullish()+ .describe(+ 'Only this environment, matched without case on deployment.environment.name, then deployment.environment, then env.'+ ),+ serviceNames: zod+ .array(zod.string())+ .max(tracingSpansServiceHealthCreateBodyQueryOneServiceNamesMax)+ .optional()+ .describe('Only these services.'),+ spanScope: zod+ .enum(['entry', 'all'])+ .describe('\* `entry` - Entry\n\* `all` - All')+ .default(tracingSpansServiceHealthCreateBodyQueryOneSpanScopeDefault)+ .describe(+ 'entry (default) counts root, server and consumer spans, one per request. all counts every span.\n\n\* `entry` - Entry\n\* `all` - All'+ ),+ orderBy: zod+ .enum(['status', 'error_rate', 'p95', 'throughput', 'error_count'])+ .describe(+ '\* `status` - Status\n\* `error_rate` - Error Rate\n\* `p95` - P95\n\* `throughput` - Throughput\n\* `error_count` - Error Count'+ )+ .default(tracingSpansServiceHealthCreateBodyQueryOneOrderByDefault)+ .describe(+ 'status (default) puts silent, regressed and degraded first. The others sort high to low.\n\n\* `status` - Status\n\* `error_rate` - Error Rate\n\* `p95` - P95\n\* `throughput` - Throughput\n\* `error_count` - Error Count'+ ),+ limit: zod+ .number()+ .min(1)+ .max(tracingSpansServiceHealthCreateBodyQueryOneLimitMax)+ .default(tracingSpansServiceHealthCreateBodyQueryOneLimitDefault)+ .describe('Max service rows, 1-200.'),+ })+ .describe('The service health query. Every field is optional.'),+ baselineDateRange: zod+ .object({+ date_from: zod.string().nullish().describe('Relative (-30m, -6h, -7d) or ISO 8601.'),+ date_to: zod.string().nullish().describe('Same format. Omit or null for \"now\".'),+ })+ .optional()+ .describe('An explicit baseline window. Overrides query.compare.'),+})+export const TracingSpansSparklineCreateParams = () => zod.object({project_id: zod.string()@@ -1077,6 +1173,15 @@ export const TracingSpansValuesRetrieveQueryParams = () => zod.object({.describe('Type of attribute: \"span\" for built-in span fields (e.g. name), \"span_attribute\" for span-level attributes, \"span_resource_attribute\" for resource-level attributes.\n\n\* `span` - span\n\* `span_attribute` - span_attribute\n\* `span_resource_attribute` - span_resource_attribute'),+ dateRange: zod+ .string()+ .min(1)+ .optional()+ .describe(+ 'JSON-encoded date range object, e.g. {\"date_from\": \"-1h\"}. Malformed JSON returns 400. Or pass date_from and date_to.'+ ),+ date_from: zod.string().min(1).optional().describe('Range start: -1h, -7d or ISO 8601. Defaults to -1h.'),+ date_to: zod.string().min(1).optional().describe('Range end, same format. Omit for \"now\".'),key: zod.string().min(1).describe('The attribute key to get values for.'),limit: zod.number()diff --git a/services/mcp/src/tools/generated/tracing.ts b/services/mcp/src/tools/generated/tracing.tsindex 22252070771d..a680f7355bdb 100644--- a/services/mcp/src/tools/generated/tracing.ts+++ b/services/mcp/src/tools/generated/tracing.ts@@ -50,6 +50,9 @@ const apmAttributeValuesList = (): ToolBase<ReturnType<typeof ApmAttributeValuespath: `/api/projects/${encodeURIComponent(String(projectId))}/tracing/spans/values/`,query: {attribute_type: params.attribute_type,+ dateRange: params.dateRange,+ date_from: params.date_from,+ date_to: params.date_to,key: params.key,limit: params.limit,offset: params.offset,@@ -79,6 +82,9 @@ const apmAttributesList = (): ToolBase<path: `/api/projects/${encodeURIComponent(String(projectId))}/tracing/spans/attributes/`,query: {attribute_type: params.attribute_type,+ dateRange: params.dateRange,+ date_from: params.date_from,+ date_to: params.date_to,limit: params.limit,offset: params.offset,search: params.search,@@ -90,6 +96,32 @@ const apmAttributesList = (): ToolBase<},})+const ApmServiceHealthSchema = () => {+ const TracingSpansServiceHealthCreateBody = orvalSchemas.TracingSpansServiceHealthCreateBody()+ return TracingSpansServiceHealthCreateBody+}++const apmServiceHealth = (): ToolBase<ReturnType<typeof ApmServiceHealthSchema>, Schemas.ServiceHealthResponse> => ({+ name: 'apm-service-health',+ schema: ApmServiceHealthSchema(),+ handler: async (context: Context, params: z.infer<ReturnType<typeof ApmServiceHealthSchema>>) => {+ const projectId = await context.stateManager.getProjectId()+ const body: Record<string, unknown> = {}+ if (params.query !== undefined) {+ body['query'] = params.query+ }+ if (params.baselineDateRange !== undefined) {+ body['baselineDateRange'] = params.baselineDateRange+ }+ const result = await context.api.request<Schemas.ServiceHealthResponse>({+ method: 'POST',+ path: `/api/projects/${encodeURIComponent(String(projectId))}/tracing/spans/service_health/`,+ body,+ })+ return result+ },+})+const ApmServicesListSchema = () => {const TracingSpansServiceNamesRetrieveQueryParams = orvalSchemas.TracingSpansServiceNamesRetrieveQueryParams()return TracingSpansServiceNamesRetrieveQueryParams.extend({@@ -112,6 +144,8 @@ const apmServicesList = (): ToolBase<path: `/api/projects/${encodeURIComponent(String(projectId))}/tracing/spans/service-names/`,query: {dateRange: params.dateRange,+ date_from: params.date_from,+ date_to: params.date_to,search: params.search,},})@@ -166,7 +200,7 @@ const apmSpansCount = (): ToolBase<ReturnType<typeof ApmSpansCountSchema>, Schempath: `/api/projects/${encodeURIComponent(String(projectId))}/tracing/spans/count/`,body,})- const filtered = pickResponseFields(result, ['count']) as typeof result+ const filtered = pickResponseFields(result, ['count', 'traceCount']) as typeof resultreturn filtered},})@@ -337,6 +371,7 @@ export const GENERATED_TOOLS: Record<string, () => ToolBase<ZodObjectAny>> = {'apm-attribute-breakdown': apmAttributeBreakdown,'apm-attribute-values-list': apmAttributeValuesList,'apm-attributes-list': apmAttributesList,+ 'apm-service-health': apmServiceHealth,'apm-services-list': apmServicesList,'apm-spans-aggregate': apmSpansAggregate,'apm-spans-count': apmSpansCount,diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attribute-values-list.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attribute-values-list.jsonindex 7efc1fbccb6d..581cd6be2fce 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attribute-values-list.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attribute-values-list.json@@ -6,6 +6,21 @@"enum": ["span", "span_attribute", "span_resource_attribute"],"type": "string"},+ "date_from": {+ "description": "Range start: -1h, -7d or ISO 8601. Defaults to -1h.",+ "minLength": 1,+ "type": "string"+ },+ "date_to": {+ "description": "Range end, same format. Omit for \"now\".",+ "minLength": 1,+ "type": "string"+ },+ "dateRange": {+ "description": "JSON-encoded date range object, e.g. {\"date_from\": \"-1h\"}. Malformed JSON returns 400. Or pass date_from and date_to.",+ "minLength": 1,+ "type": "string"+ },"key": {"description": "The attribute key to get values for.","minLength": 1,diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attributes-list.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attributes-list.jsonindex f71dd9fb3440..454dc07a66e3 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attributes-list.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-attributes-list.json@@ -6,6 +6,21 @@"enum": ["span_attribute", "span_resource_attribute"],"type": "string"},+ "date_from": {+ "description": "Range start: -1h, -7d or ISO 8601. Defaults to -1h.",+ "minLength": 1,+ "type": "string"+ },+ "date_to": {+ "description": "Range end, same format. Omit for \"now\".",+ "minLength": 1,+ "type": "string"+ },+ "dateRange": {+ "description": "JSON-encoded date range object, e.g. {\"date_from\": \"-1h\"}. Malformed JSON returns 400. Or pass date_from and date_to.",+ "minLength": 1,+ "type": "string"+ },"limit": {"description": "Max results (default: 100).","maximum": 100,diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-service-health.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-service-health.jsonnew file mode 100644index 000000000000..5505e5a5b7a7--- /dev/null+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-service-health.json@@ -0,0 +1,113 @@+{+ "$schema": "https://json-schema.org/draft/2020-12/schema",+ "properties": {+ "baselineDateRange": {+ "description": "An explicit baseline window. Overrides query.compare.",+ "properties": {+ "date_from": {+ "anyOf": [+ {+ "type": "string"+ },+ {+ "type": "null"+ }+ ],+ "description": "Relative (-30m, -6h, -7d) or ISO 8601."+ },+ "date_to": {+ "anyOf": [+ {+ "type": "string"+ },+ {+ "type": "null"+ }+ ],+ "description": "Same format. Omit or null for \"now\"."+ }+ },+ "type": "object"+ },+ "query": {+ "description": "The service health query. Every field is optional.",+ "properties": {+ "compare": {+ "default": "previous_period",+ "description": "The baseline. previous_period (default) is the same length just before the window. previous_day and previous_week shift the window back 1 or 7 days. none skips it. baselineDateRange overrides it.\n\n* `none` - None\n* `previous_period` - Previous Period\n* `previous_day` - Previous Day\n* `previous_week` - Previous Week",+ "enum": ["none", "previous_period", "previous_day", "previous_week"],+ "type": "string"+ },+ "dateRange": {+ "description": "The window to score. Defaults to the last hour.",+ "properties": {+ "date_from": {+ "anyOf": [+ {+ "type": "string"+ },+ {+ "type": "null"+ }+ ],+ "description": "Relative (-30m, -6h, -7d) or ISO 8601."+ },+ "date_to": {+ "anyOf": [+ {+ "type": "string"+ },+ {+ "type": "null"+ }+ ],+ "description": "Same format. Omit or null for \"now\"."+ }+ },+ "type": "object"+ },+ "environment": {+ "anyOf": [+ {+ "type": "string"+ },+ {+ "type": "null"+ }+ ],+ "description": "Only this environment, matched without case on deployment.environment.name, then deployment.environment, then env."+ },+ "limit": {+ "default": 25,+ "description": "Max service rows, 1-200.",+ "maximum": 200,+ "minimum": 1,+ "type": "number"+ },+ "orderBy": {+ "default": "status",+ "description": "status (default) puts silent, regressed and degraded first. The others sort high to low.\n\n* `status` - Status\n* `error_rate` - Error Rate\n* `p95` - P95\n* `throughput` - Throughput\n* `error_count` - Error Count",+ "enum": ["status", "error_rate", "p95", "throughput", "error_count"],+ "type": "string"+ },+ "serviceNames": {+ "description": "Only these services.",+ "items": {+ "type": "string"+ },+ "maxItems": 100,+ "type": "array"+ },+ "spanScope": {+ "default": "entry",+ "description": "entry (default) counts root, server and consumer spans, one per request. all counts every span.\n\n* `entry` - Entry\n* `all` - All",+ "enum": ["entry", "all"],+ "type": "string"+ }+ },+ "type": "object"+ }+ },+ "required": ["query"],+ "type": "object"+}diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-services-list.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-services-list.jsonindex 2520e54f4632..39129bfdcdf8 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-services-list.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/apm-services-list.json@@ -1,8 +1,18 @@{"$schema": "https://json-schema.org/draft/2020-12/schema","properties": {+ "date_from": {+ "description": "Range start: -1h, -7d or ISO 8601. Defaults to -1h.",+ "minLength": 1,+ "type": "string"+ },+ "date_to": {+ "description": "Range end, same format. Omit for \"now\".",+ "minLength": 1,+ "type": "string"+ },"dateRange": {- "description": "JSON-encoded date range, e.g. '{\"date_from\": \"-1h\"}'.",+ "description": "JSON-encoded date range object, e.g. {\"date_from\": \"-1h\"}. Malformed JSON returns 400. Or pass date_from and date_to.","minLength": 1,"type": "string"},