:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(alerts-platform): record tick duration, keys and schedule skips (#125805)
Jimmy Zhucd097eb0126b
diff --git a/docs/internal/alerts-platform-temporal.md b/docs/internal/alerts-platform-temporal.mdindex 3d205f7ee325..daf13a4da440 100644--- a/docs/internal/alerts-platform-temporal.md+++ b/docs/internal/alerts-platform-temporal.md@@ -480,19 +480,23 @@ The path emits through Temporal's own meter, so every series carries the worker,the runtime attaches. `products/alerts_platform/backend/temporal/metrics.py` holds them and a source reaches them through`facade/platform_metrics.py`.-| Metric | What it answers |-| --------------------------------------------------------- | ----------------------------------------------------------------- |-| `alerts_platform_checks_total{source,outcome}` | How many checks the platform decided, and what they decided |-| `alerts_platform_state_transitions_total{source,from,to}` | Which transitions it reached |-| `alerts_platform_deliveries_previewed_total{source}` | How many deliveries it recorded instead of sending |-| `alerts_platform_deliveries_deferred_total{source}` | How many the payload bound left for a later tick |-| `alerts_platform_outcomes_recorded_total` | How many decisions reached the tables |-| `alerts_platform_batch_duration_ms{source}` | What one batch key costs |-| `alerts_platform_scheduler_lag_ms{source}` | How far past its due time a check was evaluated |-| `alerts_platform_discovery_omitted_total{source}` | How many due batch keys a tick left out under the discovery limit |-| `alerts_platform_checks_deferred_total{source}` | How many checks a full evaluation pool turned away |-| `alerts_platform_checks_unfinished_total{source}` | How many admitted insight checks reached no outcome |-| `alerts_platform_history_rows_dropped_total{source}` | How many check rows a failed history write lost |+| Metric | What it answers |+| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |+| `alerts_platform_checks_total{source,outcome}` | How many checks the platform decided, and what they decided |+| `alerts_platform_state_transitions_total{source,from,to}` | Which transitions it reached |+| `alerts_platform_deliveries_previewed_total{source}` | How many deliveries it recorded instead of sending |+| `alerts_platform_deliveries_deferred_total{source}` | How many the payload bound left for a later tick |+| `alerts_platform_outcomes_recorded_total` | How many decisions reached the tables |+| `alerts_platform_batch_duration_ms{source}` | What one batch key costs |+| `alerts_platform_scheduler_lag_ms{source}` | How far past its due time a check was evaluated |+| `alerts_platform_discovery_omitted_total{source}` | How many due batch keys a tick left out under the discovery limit |+| `alerts_platform_tick_duration_ms{deadline_reached}` | How long a tick ran from its scheduled minute. Past 60 s, the schedule skips the next tick |+| `alerts_platform_tick_start_delay_ms` | How late a tick's first workflow task ran after its scheduled minute |+| `alerts_platform_tick_keys` | How many batch keys a tick's discovery found due, including those its limits left out |+| `alerts_platform_tick_schedule_actions{outcome}` | The tick schedule's lifetime counts of started, overlap-skipped and late-dropped ticks, pushed every 5 minutes. A rise in `skipped_overlap` or `missed_catchup` is a lost tick |+| `alerts_platform_checks_deferred_total{source}` | How many checks a full evaluation pool turned away |+| `alerts_platform_checks_unfinished_total{source}` | How many admitted insight checks reached no outcome |+| `alerts_platform_history_rows_dropped_total{source}` | How many check rows a failed history write lost |Histogram buckets are registered in `posthog/temporal/common/worker.py`; a histogram missing from`ALERTS_PLATFORM_LATENCY_HISTOGRAM_METRICS` gets Prometheus defaults instead.diff --git a/posthog/temporal/common/worker.py b/posthog/temporal/common/worker.pyindex 71ba1ebf40b2..67877e7c9996 100644--- a/posthog/temporal/common/worker.py+++ b/posthog/temporal/common/worker.py@@ -72,6 +72,10 @@ from posthog.temporal.usage_report.metrics import (from products.alerts_platform.backend.facade.temporal import (ALERTS_PLATFORM_LATENCY_HISTOGRAM_BUCKETS,ALERTS_PLATFORM_LATENCY_HISTOGRAM_METRICS,+ ALERTS_PLATFORM_TICK_HISTOGRAM_BUCKETS,+ ALERTS_PLATFORM_TICK_HISTOGRAM_METRICS,+ ALERTS_PLATFORM_TICK_KEYS_BUCKETS,+ ALERTS_PLATFORM_TICK_KEYS_METRICS,AlertsPlatformTelemetryInterceptor,)from products.batch_exports.backend.facade.temporal import BatchExportsMetricsInterceptor@@ -410,6 +414,8 @@ async def create_worker(itertools.repeat(ALERTS_PLATFORM_LATENCY_HISTOGRAM_BUCKETS),))+ | dict(zip(ALERTS_PLATFORM_TICK_HISTOGRAM_METRICS, itertools.repeat(ALERTS_PLATFORM_TICK_HISTOGRAM_BUCKETS)))+ | dict(zip(ALERTS_PLATFORM_TICK_KEYS_METRICS, itertools.repeat(ALERTS_PLATFORM_TICK_KEYS_BUCKETS)))| dict(zip(LOGS_ALERTING_LATENCY_HISTOGRAM_METRICS, itertools.repeat(LOGS_ALERTING_LATENCY_HISTOGRAM_BUCKETS)))| dict(zip(LOGS_ALERTING_COUNT_HISTOGRAM_METRICS, itertools.repeat(LOGS_ALERTING_COUNT_HISTOGRAM_BUCKETS)))| dict(diff --git a/products/alerts_platform/backend/facade/temporal.py b/products/alerts_platform/backend/facade/temporal.pyindex b0aab84288c4..6b5a689597d3 100644--- a/products/alerts_platform/backend/facade/temporal.py+++ b/products/alerts_platform/backend/facade/temporal.py@@ -1,6 +1,10 @@from products.alerts_platform.backend.temporal.metrics import (ALERTS_PLATFORM_LATENCY_HISTOGRAM_BUCKETS,ALERTS_PLATFORM_LATENCY_HISTOGRAM_METRICS,+ ALERTS_PLATFORM_TICK_HISTOGRAM_BUCKETS,+ ALERTS_PLATFORM_TICK_HISTOGRAM_METRICS,+ ALERTS_PLATFORM_TICK_KEYS_BUCKETS,+ ALERTS_PLATFORM_TICK_KEYS_METRICS,)from products.alerts_platform.backend.temporal.schedule import (create_alerts_platform_expired_holds_schedule,@@ -23,6 +27,10 @@ from products.alerts_platform.backend.temporal.workflows import (__all__ = ["ALERTS_PLATFORM_LATENCY_HISTOGRAM_BUCKETS","ALERTS_PLATFORM_LATENCY_HISTOGRAM_METRICS",+ "ALERTS_PLATFORM_TICK_HISTOGRAM_BUCKETS",+ "ALERTS_PLATFORM_TICK_HISTOGRAM_METRICS",+ "ALERTS_PLATFORM_TICK_KEYS_BUCKETS",+ "ALERTS_PLATFORM_TICK_KEYS_METRICS","DELIVERY_ACTIVITIES","DELIVERY_EXECUTION_TIMEOUT","DELIVERY_WORKFLOWS",diff --git a/products/alerts_platform/backend/temporal/inventory_metrics.py b/products/alerts_platform/backend/temporal/inventory_metrics.pyindex 2f034b34ca04..a1168881476f 100644--- a/products/alerts_platform/backend/temporal/inventory_metrics.py+++ b/products/alerts_platform/backend/temporal/inventory_metrics.py@@ -9,6 +9,7 @@ from collections.abc import Sequencefrom prometheus_client import Gauge+from posthog.dataclasses import frozenfrom posthog.metrics import pushed_metrics_registryfrom products.alerts_platform.backend.logic.inventory import SourceInventory@@ -18,6 +19,30 @@ from products.alerts_platform.backend.logic.inventory import SourceInventoryPUSH_TIMEOUT_SECONDS = 5+@frozen+class TickScheduleRuns:+ """The tick schedule's lifetime counts, as Temporal keeps them for this schedule alone."""++ started: int+ skipped_overlap: int+ missed_catchup: int+++def push_tick_schedule_runs(schedule_runs: TickScheduleRuns) -> None:+ """A group of its own, so a failed inventory count neither holds these back nor replaces them."""+ with pushed_metrics_registry("alerts_platform_tick_schedule", timeout=PUSH_TIMEOUT_SECONDS) as registry:+ # Lifetime counts, so a rise between two pushes is a tick the schedule skipped or dropped.+ runs = Gauge(+ "alerts_platform_tick_schedule_actions",+ "Ticks the platform's schedule started, skipped for overlap, or dropped for starting late",+ ["outcome"],+ registry=registry,+ )+ runs.labels(outcome="started").set(schedule_runs.started)+ runs.labels(outcome="skipped_overlap").set(schedule_runs.skipped_overlap)+ runs.labels(outcome="missed_catchup").set(schedule_runs.missed_catchup)++def push_inventory(inventories: Sequence[SourceInventory]) -> None:with pushed_metrics_registry("alerts_platform_inventory", timeout=PUSH_TIMEOUT_SECONDS) as registry:configurations = Gauge(diff --git a/products/alerts_platform/backend/temporal/metrics.py b/products/alerts_platform/backend/temporal/metrics.pyindex 8e69c68939c3..54c81e218024 100644--- a/products/alerts_platform/backend/temporal/metrics.py+++ b/products/alerts_platform/backend/temporal/metrics.py@@ -39,6 +39,50 @@ ALERTS_PLATFORM_LATENCY_HISTOGRAM_BUCKETS = []+# A tick must end inside its minute, so these buckets resolve the seconds up to and past 60.+ALERTS_PLATFORM_TICK_HISTOGRAM_METRICS = (+ "alerts_platform_tick_duration_ms",+ "alerts_platform_tick_start_delay_ms",+)++ALERTS_PLATFORM_TICK_HISTOGRAM_BUCKETS = [+ 100.0,+ 500.0,+ 1_000.0,+ 2_000.0,+ 5_000.0,+ 10_000.0,+ 15_000.0,+ 20_000.0,+ 30_000.0,+ 40_000.0,+ 50_000.0,+ 60_000.0,+ 90_000.0,+ 120_000.0,+]++ALERTS_PLATFORM_TICK_KEYS_METRICS = ("alerts_platform_tick_keys",)++ALERTS_PLATFORM_TICK_KEYS_BUCKETS = [+ 0.0,+ 50.0,+ 100.0,+ 200.0,+ 300.0,+ 400.0,+ 500.0,+ 600.0,+ 800.0,+ 1_000.0,+ 1_250.0,+ 1_500.0,+ 2_000.0,+ 3_000.0,+ 5_000.0,+]++def safe_record(fn: Callable[..., object], *args: object, **kwargs: object) -> None:"""Best-effort metric recording. A batch that decided correctly must not fail on telemetry."""try:@@ -273,6 +317,26 @@ def record_delivery_lag(source: str, provider: str, lag_ms: int) -> None:).record(dt.timedelta(milliseconds=lag_ms))+def record_tick_keys(keys: int) -> None:+ """Batch keys one tick's discovery found due, the ones its limits left out included. Called+ from workflow code; the workflow meter emits nothing during replay."""+ get_metric_meter().create_histogram(+ name=ALERTS_PLATFORM_TICK_KEYS_METRICS[0],+ description="Batch keys one tick's discovery found due, including those its limits left out",+ ).record(keys)+++def record_tick_end(*, duration: dt.timedelta, deadline_reached: bool) -> None:+ """One finished tick, measured from its scheduled minute. A tick that runs past its minute+ makes the schedule skip the next one, so the duration's distance from 60 seconds is the+ headroom left. Called from workflow code; the workflow meter emits nothing during replay."""+ get_metric_meter({"deadline_reached": str(deadline_reached).lower()}).create_histogram_timedelta(+ name="alerts_platform_tick_duration_ms",+ description="Time from a tick's scheduled minute to the end of its last dispatch page",+ unit="ms",+ ).record(duration)++def record_scheduler_lag(source: str, lag_ms: int) -> None:"""How far past its due time a check was evaluated. Rising lag means discovery's bound isbiting, because a key it omits grows more overdue until a later tick takes it."""diff --git a/products/alerts_platform/backend/temporal/workflows.py b/products/alerts_platform/backend/temporal/workflows.pyindex 4273a8016396..37f52178b885 100644--- a/products/alerts_platform/backend/temporal/workflows.py+++ b/products/alerts_platform/backend/temporal/workflows.py@@ -23,6 +23,8 @@ from products.alerts_platform.backend.temporal.metrics import (increment_deliveries_previewed,increment_discovery_omitted,increment_holds_expired,+ record_tick_end,+ record_tick_keys,safe_record,)from products.alerts_platform.backend.temporal.outcomes import alerts_platform_record_outcomes_activity@@ -36,6 +38,8 @@ with workflow.unsafe.imports_passed_through():from asgiref.sync import sync_to_asyncfrom posthog.sync import database_sync_to_async_pool+ from posthog.temporal.common.metrics import record_schedule_to_start_latency+ from posthog.temporal.common.schedule import a_describe_schedulefrom products.alerts_platform.backend.delivery.evaluation import deliver_evaluationfrom products.alerts_platform.backend.delivery.thread_store import PENDING_CLAIM_TTL, ThreadBusy@@ -56,8 +60,13 @@ with workflow.unsafe.imports_passed_through():from products.alerts_platform.backend.logic import delivery_holdsfrom products.alerts_platform.backend.logic.demand import discover_demandfrom products.alerts_platform.backend.logic.inventory import count_inventory- from products.alerts_platform.backend.temporal.inventory_metrics import push_inventory+ from products.alerts_platform.backend.temporal.inventory_metrics import (+ TickScheduleRuns,+ push_inventory,+ push_tick_schedule_runs,+ )from products.alerts_platform.backend.temporal.postgres import check_postgres_connection+ from products.alerts_platform.backend.temporal.schedule import SCHEDULE_IDfrom products.alerts_platform.backend.temporal.sources import SOURCE_BINDINGS, discovery_limits@@ -103,6 +112,8 @@ CONCURRENT_EVALUATION_STARTS_PATCH = "alerts-platform-concurrent-evaluation-starBUSY_WAIT_LIMIT = PENDING_CLAIM_TTL + BUSY_RETRY_DELAY# The last attempt can start one attempt and one delay past the limit.DELIVERY_EXECUTION_TIMEOUT = 3 * DELIVERY_SCHEDULE_TO_CLOSE + BUSY_WAIT_LIMIT + BUSY_RETRY_DELAY+# Leaves most of the inventory activity's 20 seconds to the inventory count after a stalled describe.+SCHEDULE_DESCRIBE_TIMEOUT = dt.timedelta(seconds=5)@frozen@@ -119,16 +130,34 @@ async def alerts_platform_discover_demand_activity(inputs: DemandDiscoveryInputsreturn demand+async def _tick_schedule_runs() -> TickScheduleRuns | None:+ """The tick schedule's own run counts. Temporal's namespace metrics do not say which schedule skipped."""+ try:+ async with asyncio.timeout(SCHEDULE_DESCRIBE_TIMEOUT.total_seconds()):+ info = (await a_describe_schedule(activity.client(), SCHEDULE_ID)).info+ except Exception as error:+ await LOGGER.awarning("alerts_platform_schedule_describe_failed", error=str(error))+ return None+ return TickScheduleRuns(+ started=info.num_actions,+ skipped_overlap=info.num_actions_skipped_overlap,+ missed_catchup=info.num_actions_missed_catchup_window,+ )++@activity.defnasync def alerts_platform_record_inventory_activity() -> None:"""A failed count is logged rather than raised, so a slow count never reads as a failed workflow."""+ schedule_runs = await _tick_schedule_runs()+ # A failed push is logged and captured inside `push_registry`, never raised.+ if schedule_runs is not None:+ await asyncio.to_thread(push_tick_schedule_runs, schedule_runs)now = dt.datetime.now(dt.UTC)try:inventories = await database_sync_to_async_pool(count_inventory)(now)except Exception as error:await LOGGER.awarning("alerts_platform_inventory_failed", error=str(error))return- # A failed push is logged and captured inside `push_registry`, never raised.await asyncio.to_thread(push_inventory, inventories)@@ -532,6 +561,26 @@ def _should_continue_as_new() -> bool:return workflow.info().is_continue_as_new_suggested()+def _finished_tick(result: OrchestrateResult, cutoff_iso: str) -> OrchestrateResult:+ """Records how long the tick took from its scheduled minute.++ The tick logs no other line, so this one is how an investigation reads a single tick's numbers.+ """+ duration = workflow.now() - dt.datetime.fromisoformat(cutoff_iso)+ safe_record(record_tick_end, duration=duration, deadline_reached=result.deadline_reached)+ # The log formatter drops `extra` fields, so the numbers go in the message.+ workflow.logger.info(+ "Alerts platform tick finished: scheduled_at=%s duration_ms=%d dispatched=%d remaining=%d pages=%d deadline_reached=%s",+ cutoff_iso,+ int(duration.total_seconds() * 1000),+ sum(page.dispatched for page in result.pages),+ result.remaining,+ len(result.pages),+ result.deadline_reached,+ )+ return result++@workflow.defn(name="alerts-platform-orchestrate")class AlertsPlatformOrchestrateWorkflow(PostHogWorkflow):"""One minute tick. Discovers demand once, then pages source dispatchers until the demand is@@ -550,6 +599,12 @@ class AlertsPlatformOrchestrateWorkflow(PostHogWorkflow):SearchAttributeKey.for_datetime("TemporalScheduledStartTime"), info.workflow_start_time)now = workflow.now()+ record_schedule_to_start_latency(+ "alerts_platform_tick_start_delay_ms",+ scheduled_time=cutoff,+ started_time=info.start_time,+ description="Time from a tick's scheduled minute to its first workflow task",+ )hard_stop = info.execution_timeout or (TICK_DISPATCH_BUDGET + TICK_HARD_STOP_MARGIN)inputs = replace(inputs,@@ -572,6 +627,7 @@ class AlertsPlatformOrchestrateWorkflow(PostHogWorkflow):)demand = discovered.batch_keys_by_sourceinputs = replace(inputs, omitted=sum(discovered.omitted_by_source.values()))+ safe_record(record_tick_keys, sum(len(keys) for keys in demand.values()) + inputs.omitted)else:demand = inputs.demand@@ -589,7 +645,9 @@ class AlertsPlatformOrchestrateWorkflow(PostHogWorkflow):+ inputs.omitted+ sum(recorded.undispatched for recorded in pages))- return OrchestrateResult(pages=pages, remaining=remaining, deadline_reached=True)+ return _finished_tick(+ OrchestrateResult(pages=pages, remaining=remaining, deadline_reached=True), cutoff_iso+ )# One list, so the settled results below can be read back against the source that# produced each one.paged = sorted(demand.items())@@ -650,10 +708,13 @@ class AlertsPlatformOrchestrateWorkflow(PostHogWorkflow):if demand and _should_continue_as_new():workflow.continue_as_new(replace(inputs, page=page, demand=demand, pages=pages))- return OrchestrateResult(- pages=pages,- remaining=inputs.omitted + sum(recorded.undispatched for recorded in pages),- deadline_reached=False,+ return _finished_tick(+ OrchestrateResult(+ pages=pages,+ remaining=inputs.omitted + sum(recorded.undispatched for recorded in pages),+ deadline_reached=False,+ ),+ cutoff_iso,)diff --git a/products/alerts_platform/backend/tests/test_inventory.py b/products/alerts_platform/backend/tests/test_inventory.pyindex d7d7e85391b5..2d6e3387050c 100644--- a/products/alerts_platform/backend/tests/test_inventory.py+++ b/products/alerts_platform/backend/tests/test_inventory.py@@ -18,7 +18,11 @@ from products.alerts_platform.backend.logic.inventory import (count_inventory,)from products.alerts_platform.backend.models import PlatformAlert, PlatformAlertConfiguration-from products.alerts_platform.backend.temporal.inventory_metrics import push_inventory+from products.alerts_platform.backend.temporal.inventory_metrics import (+ TickScheduleRuns,+ push_inventory,+ push_tick_schedule_runs,+)NOW = datetime(2026, 10, 23, 12, tzinfo=UTC)LATER = NOW + timedelta(hours=1)@@ -139,3 +143,15 @@ def test_push_replaces_the_group_with_the_dashboard_labels() -> None:assert (registry.get_sample_value("alerts_platform_alerts", {"source": "logs", "state": "firing", "muted": "true"}) == 2)+++def test_schedule_runs_push_to_a_group_of_their_own() -> None:+ with patch("posthog.metrics.push_registry") as push:+ push_tick_schedule_runs(TickScheduleRuns(started=120, skipped_overlap=1, missed_catchup=3))++ assert push.call_args.args[0] == "alerts_platform_tick_schedule"+ registry = push.call_args.args[1]+ assert {+ outcome: registry.get_sample_value("alerts_platform_tick_schedule_actions", {"outcome": outcome})+ for outcome in ("started", "skipped_overlap", "missed_catchup")+ } == {"started": 120, "skipped_overlap": 1, "missed_catchup": 3}diff --git a/products/alerts_platform/backend/tests/test_temporal_workflows.py b/products/alerts_platform/backend/tests/test_temporal_workflows.pyindex e2138a1888f0..d3e58846371d 100644--- a/products/alerts_platform/backend/tests/test_temporal_workflows.py+++ b/products/alerts_platform/backend/tests/test_temporal_workflows.py@@ -663,6 +663,7 @@ async def test_discovery_uses_scheduled_cutoff_or_manual_start(scheduled: bool))info = MagicMock(workflow_start_time=actual_start,+ start_time=actual_start,typed_search_attributes=attributes,workflow_id="tick",run_id="run",@@ -675,6 +676,7 @@ async def test_discovery_uses_scheduled_cutoff_or_manual_start(scheduled: bool)workflow, "execute_activity", AsyncMock(return_value=AlertDemand(batch_keys_by_source={}))) as discover,patch.object(workflow, "start_child_workflow", AsyncMock()) as dispatch,+ patch.object(workflow, "logger"),):result = await AlertsPlatformOrchestrateWorkflow().run(OrchestrateInputs())assert discover.await_args is not None