Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(alerts-platform): record tick duration, keys and schedule skips (#125805)

Jimmy Zhucd097eb0126b
diff --git a/docs/internal/alerts-platform-temporal.md b/docs/internal/alerts-platform-temporal.md
index 3d205f7ee325..daf13a4da440 100644
--- a/docs/internal/alerts-platform-temporal.md
+++ b/docs/internal/alerts-platform-temporal.md
@@ -480,19 +480,23 @@ The path emits through Temporal's own meter, so every series carries the worker,
the runtime attaches. `products/alerts_platform/backend/temporal/metrics.py` holds them and a source reaches them through
`facade/platform_metrics.py`.
-| Metric | What it answers |
-| --------------------------------------------------------- | ----------------------------------------------------------------- |
-| `alerts_platform_checks_total{source,outcome}` | How many checks the platform decided, and what they decided |
-| `alerts_platform_state_transitions_total{source,from,to}` | Which transitions it reached |
-| `alerts_platform_deliveries_previewed_total{source}` | How many deliveries it recorded instead of sending |
-| `alerts_platform_deliveries_deferred_total{source}` | How many the payload bound left for a later tick |
-| `alerts_platform_outcomes_recorded_total` | How many decisions reached the tables |
-| `alerts_platform_batch_duration_ms{source}` | What one batch key costs |
-| `alerts_platform_scheduler_lag_ms{source}` | How far past its due time a check was evaluated |
-| `alerts_platform_discovery_omitted_total{source}` | How many due batch keys a tick left out under the discovery limit |
-| `alerts_platform_checks_deferred_total{source}` | How many checks a full evaluation pool turned away |
-| `alerts_platform_checks_unfinished_total{source}` | How many admitted insight checks reached no outcome |
-| `alerts_platform_history_rows_dropped_total{source}` | How many check rows a failed history write lost |
+| Metric | What it answers |
+| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `alerts_platform_checks_total{source,outcome}` | How many checks the platform decided, and what they decided |
+| `alerts_platform_state_transitions_total{source,from,to}` | Which transitions it reached |
+| `alerts_platform_deliveries_previewed_total{source}` | How many deliveries it recorded instead of sending |
+| `alerts_platform_deliveries_deferred_total{source}` | How many the payload bound left for a later tick |
+| `alerts_platform_outcomes_recorded_total` | How many decisions reached the tables |
+| `alerts_platform_batch_duration_ms{source}` | What one batch key costs |
+| `alerts_platform_scheduler_lag_ms{source}` | How far past its due time a check was evaluated |
+| `alerts_platform_discovery_omitted_total{source}` | How many due batch keys a tick left out under the discovery limit |
+| `alerts_platform_tick_duration_ms{deadline_reached}` | How long a tick ran from its scheduled minute. Past 60 s, the schedule skips the next tick |
+| `alerts_platform_tick_start_delay_ms` | How late a tick's first workflow task ran after its scheduled minute |
+| `alerts_platform_tick_keys` | How many batch keys a tick's discovery found due, including those its limits left out |
+| `alerts_platform_tick_schedule_actions{outcome}` | The tick schedule's lifetime counts of started, overlap-skipped and late-dropped ticks, pushed every 5 minutes. A rise in `skipped_overlap` or `missed_catchup` is a lost tick |
+| `alerts_platform_checks_deferred_total{source}` | How many checks a full evaluation pool turned away |
+| `alerts_platform_checks_unfinished_total{source}` | How many admitted insight checks reached no outcome |
+| `alerts_platform_history_rows_dropped_total{source}` | How many check rows a failed history write lost |
Histogram buckets are registered in `posthog/temporal/common/worker.py`; a histogram missing from
`ALERTS_PLATFORM_LATENCY_HISTOGRAM_METRICS` gets Prometheus defaults instead.
diff --git a/posthog/temporal/common/worker.py b/posthog/temporal/common/worker.py
index 71ba1ebf40b2..67877e7c9996 100644
--- a/posthog/temporal/common/worker.py
+++ b/posthog/temporal/common/worker.py
@@ -72,6 +72,10 @@ from posthog.temporal.usage_report.metrics import (
from products.alerts_platform.backend.facade.temporal import (
ALERTS_PLATFORM_LATENCY_HISTOGRAM_BUCKETS,
ALERTS_PLATFORM_LATENCY_HISTOGRAM_METRICS,
+ ALERTS_PLATFORM_TICK_HISTOGRAM_BUCKETS,
+ ALERTS_PLATFORM_TICK_HISTOGRAM_METRICS,
+ ALERTS_PLATFORM_TICK_KEYS_BUCKETS,
+ ALERTS_PLATFORM_TICK_KEYS_METRICS,
AlertsPlatformTelemetryInterceptor,
)
from products.batch_exports.backend.facade.temporal import BatchExportsMetricsInterceptor
@@ -410,6 +414,8 @@ async def create_worker(
itertools.repeat(ALERTS_PLATFORM_LATENCY_HISTOGRAM_BUCKETS),
)
)
+ | dict(zip(ALERTS_PLATFORM_TICK_HISTOGRAM_METRICS, itertools.repeat(ALERTS_PLATFORM_TICK_HISTOGRAM_BUCKETS)))
+ | dict(zip(ALERTS_PLATFORM_TICK_KEYS_METRICS, itertools.repeat(ALERTS_PLATFORM_TICK_KEYS_BUCKETS)))
| dict(zip(LOGS_ALERTING_LATENCY_HISTOGRAM_METRICS, itertools.repeat(LOGS_ALERTING_LATENCY_HISTOGRAM_BUCKETS)))
| dict(zip(LOGS_ALERTING_COUNT_HISTOGRAM_METRICS, itertools.repeat(LOGS_ALERTING_COUNT_HISTOGRAM_BUCKETS)))
| dict(
diff --git a/products/alerts_platform/backend/facade/temporal.py b/products/alerts_platform/backend/facade/temporal.py
index b0aab84288c4..6b5a689597d3 100644
--- a/products/alerts_platform/backend/facade/temporal.py
+++ b/products/alerts_platform/backend/facade/temporal.py
@@ -1,6 +1,10 @@
from products.alerts_platform.backend.temporal.metrics import (
ALERTS_PLATFORM_LATENCY_HISTOGRAM_BUCKETS,
ALERTS_PLATFORM_LATENCY_HISTOGRAM_METRICS,
+ ALERTS_PLATFORM_TICK_HISTOGRAM_BUCKETS,
+ ALERTS_PLATFORM_TICK_HISTOGRAM_METRICS,
+ ALERTS_PLATFORM_TICK_KEYS_BUCKETS,
+ ALERTS_PLATFORM_TICK_KEYS_METRICS,
)
from products.alerts_platform.backend.temporal.schedule import (
create_alerts_platform_expired_holds_schedule,
@@ -23,6 +27,10 @@ from products.alerts_platform.backend.temporal.workflows import (
__all__ = [
"ALERTS_PLATFORM_LATENCY_HISTOGRAM_BUCKETS",
"ALERTS_PLATFORM_LATENCY_HISTOGRAM_METRICS",
+ "ALERTS_PLATFORM_TICK_HISTOGRAM_BUCKETS",
+ "ALERTS_PLATFORM_TICK_HISTOGRAM_METRICS",
+ "ALERTS_PLATFORM_TICK_KEYS_BUCKETS",
+ "ALERTS_PLATFORM_TICK_KEYS_METRICS",
"DELIVERY_ACTIVITIES",
"DELIVERY_EXECUTION_TIMEOUT",
"DELIVERY_WORKFLOWS",
diff --git a/products/alerts_platform/backend/temporal/inventory_metrics.py b/products/alerts_platform/backend/temporal/inventory_metrics.py
index 2f034b34ca04..a1168881476f 100644
--- a/products/alerts_platform/backend/temporal/inventory_metrics.py
+++ b/products/alerts_platform/backend/temporal/inventory_metrics.py
@@ -9,6 +9,7 @@ from collections.abc import Sequence
from prometheus_client import Gauge
+from posthog.dataclasses import frozen
from posthog.metrics import pushed_metrics_registry
from products.alerts_platform.backend.logic.inventory import SourceInventory
@@ -18,6 +19,30 @@ from products.alerts_platform.backend.logic.inventory import SourceInventory
PUSH_TIMEOUT_SECONDS = 5
+@frozen
+class TickScheduleRuns:
+ """The tick schedule's lifetime counts, as Temporal keeps them for this schedule alone."""
+
+ started: int
+ skipped_overlap: int
+ missed_catchup: int
+
+
+def push_tick_schedule_runs(schedule_runs: TickScheduleRuns) -> None:
+ """A group of its own, so a failed inventory count neither holds these back nor replaces them."""
+ with pushed_metrics_registry("alerts_platform_tick_schedule", timeout=PUSH_TIMEOUT_SECONDS) as registry:
+ # Lifetime counts, so a rise between two pushes is a tick the schedule skipped or dropped.
+ runs = Gauge(
+ "alerts_platform_tick_schedule_actions",
+ "Ticks the platform's schedule started, skipped for overlap, or dropped for starting late",
+ ["outcome"],
+ registry=registry,
+ )
+ runs.labels(outcome="started").set(schedule_runs.started)
+ runs.labels(outcome="skipped_overlap").set(schedule_runs.skipped_overlap)
+ runs.labels(outcome="missed_catchup").set(schedule_runs.missed_catchup)
+
+
def push_inventory(inventories: Sequence[SourceInventory]) -> None:
with pushed_metrics_registry("alerts_platform_inventory", timeout=PUSH_TIMEOUT_SECONDS) as registry:
configurations = Gauge(
diff --git a/products/alerts_platform/backend/temporal/metrics.py b/products/alerts_platform/backend/temporal/metrics.py
index 8e69c68939c3..54c81e218024 100644
--- a/products/alerts_platform/backend/temporal/metrics.py
+++ b/products/alerts_platform/backend/temporal/metrics.py
@@ -39,6 +39,50 @@ ALERTS_PLATFORM_LATENCY_HISTOGRAM_BUCKETS = [
]
+# A tick must end inside its minute, so these buckets resolve the seconds up to and past 60.
+ALERTS_PLATFORM_TICK_HISTOGRAM_METRICS = (
+ "alerts_platform_tick_duration_ms",
+ "alerts_platform_tick_start_delay_ms",
+)
+
+ALERTS_PLATFORM_TICK_HISTOGRAM_BUCKETS = [
+ 100.0,
+ 500.0,
+ 1_000.0,
+ 2_000.0,
+ 5_000.0,
+ 10_000.0,
+ 15_000.0,
+ 20_000.0,
+ 30_000.0,
+ 40_000.0,
+ 50_000.0,
+ 60_000.0,
+ 90_000.0,
+ 120_000.0,
+]
+
+ALERTS_PLATFORM_TICK_KEYS_METRICS = ("alerts_platform_tick_keys",)
+
+ALERTS_PLATFORM_TICK_KEYS_BUCKETS = [
+ 0.0,
+ 50.0,
+ 100.0,
+ 200.0,
+ 300.0,
+ 400.0,
+ 500.0,
+ 600.0,
+ 800.0,
+ 1_000.0,
+ 1_250.0,
+ 1_500.0,
+ 2_000.0,
+ 3_000.0,
+ 5_000.0,
+]
+
+
def safe_record(fn: Callable[..., object], *args: object, **kwargs: object) -> None:
"""Best-effort metric recording. A batch that decided correctly must not fail on telemetry."""
try:
@@ -273,6 +317,26 @@ def record_delivery_lag(source: str, provider: str, lag_ms: int) -> None:
).record(dt.timedelta(milliseconds=lag_ms))
+def record_tick_keys(keys: int) -> None:
+ """Batch keys one tick's discovery found due, the ones its limits left out included. Called
+ from workflow code; the workflow meter emits nothing during replay."""
+ get_metric_meter().create_histogram(
+ name=ALERTS_PLATFORM_TICK_KEYS_METRICS[0],
+ description="Batch keys one tick's discovery found due, including those its limits left out",
+ ).record(keys)
+
+
+def record_tick_end(*, duration: dt.timedelta, deadline_reached: bool) -> None:
+ """One finished tick, measured from its scheduled minute. A tick that runs past its minute
+ makes the schedule skip the next one, so the duration's distance from 60 seconds is the
+ headroom left. Called from workflow code; the workflow meter emits nothing during replay."""
+ get_metric_meter({"deadline_reached": str(deadline_reached).lower()}).create_histogram_timedelta(
+ name="alerts_platform_tick_duration_ms",
+ description="Time from a tick's scheduled minute to the end of its last dispatch page",
+ unit="ms",
+ ).record(duration)
+
+
def record_scheduler_lag(source: str, lag_ms: int) -> None:
"""How far past its due time a check was evaluated. Rising lag means discovery's bound is
biting, because a key it omits grows more overdue until a later tick takes it."""
diff --git a/products/alerts_platform/backend/temporal/workflows.py b/products/alerts_platform/backend/temporal/workflows.py
index 4273a8016396..37f52178b885 100644
--- a/products/alerts_platform/backend/temporal/workflows.py
+++ b/products/alerts_platform/backend/temporal/workflows.py
@@ -23,6 +23,8 @@ from products.alerts_platform.backend.temporal.metrics import (
increment_deliveries_previewed,
increment_discovery_omitted,
increment_holds_expired,
+ record_tick_end,
+ record_tick_keys,
safe_record,
)
from products.alerts_platform.backend.temporal.outcomes import alerts_platform_record_outcomes_activity
@@ -36,6 +38,8 @@ with workflow.unsafe.imports_passed_through():
from asgiref.sync import sync_to_async
from posthog.sync import database_sync_to_async_pool
+ from posthog.temporal.common.metrics import record_schedule_to_start_latency
+ from posthog.temporal.common.schedule import a_describe_schedule
from products.alerts_platform.backend.delivery.evaluation import deliver_evaluation
from products.alerts_platform.backend.delivery.thread_store import PENDING_CLAIM_TTL, ThreadBusy
@@ -56,8 +60,13 @@ with workflow.unsafe.imports_passed_through():
from products.alerts_platform.backend.logic import delivery_holds
from products.alerts_platform.backend.logic.demand import discover_demand
from products.alerts_platform.backend.logic.inventory import count_inventory
- from products.alerts_platform.backend.temporal.inventory_metrics import push_inventory
+ from products.alerts_platform.backend.temporal.inventory_metrics import (
+ TickScheduleRuns,
+ push_inventory,
+ push_tick_schedule_runs,
+ )
from products.alerts_platform.backend.temporal.postgres import check_postgres_connection
+ from products.alerts_platform.backend.temporal.schedule import SCHEDULE_ID
from products.alerts_platform.backend.temporal.sources import SOURCE_BINDINGS, discovery_limits
@@ -103,6 +112,8 @@ CONCURRENT_EVALUATION_STARTS_PATCH = "alerts-platform-concurrent-evaluation-star
BUSY_WAIT_LIMIT = PENDING_CLAIM_TTL + BUSY_RETRY_DELAY
# The last attempt can start one attempt and one delay past the limit.
DELIVERY_EXECUTION_TIMEOUT = 3 * DELIVERY_SCHEDULE_TO_CLOSE + BUSY_WAIT_LIMIT + BUSY_RETRY_DELAY
+# Leaves most of the inventory activity's 20 seconds to the inventory count after a stalled describe.
+SCHEDULE_DESCRIBE_TIMEOUT = dt.timedelta(seconds=5)
@frozen
@@ -119,16 +130,34 @@ async def alerts_platform_discover_demand_activity(inputs: DemandDiscoveryInputs
return demand
+async def _tick_schedule_runs() -> TickScheduleRuns | None:
+ """The tick schedule's own run counts. Temporal's namespace metrics do not say which schedule skipped."""
+ try:
+ async with asyncio.timeout(SCHEDULE_DESCRIBE_TIMEOUT.total_seconds()):
+ info = (await a_describe_schedule(activity.client(), SCHEDULE_ID)).info
+ except Exception as error:
+ await LOGGER.awarning("alerts_platform_schedule_describe_failed", error=str(error))
+ return None
+ return TickScheduleRuns(
+ started=info.num_actions,
+ skipped_overlap=info.num_actions_skipped_overlap,
+ missed_catchup=info.num_actions_missed_catchup_window,
+ )
+
+
@activity.defn
async def alerts_platform_record_inventory_activity() -> None:
"""A failed count is logged rather than raised, so a slow count never reads as a failed workflow."""
+ schedule_runs = await _tick_schedule_runs()
+ # A failed push is logged and captured inside `push_registry`, never raised.
+ if schedule_runs is not None:
+ await asyncio.to_thread(push_tick_schedule_runs, schedule_runs)
now = dt.datetime.now(dt.UTC)
try:
inventories = await database_sync_to_async_pool(count_inventory)(now)
except Exception as error:
await LOGGER.awarning("alerts_platform_inventory_failed", error=str(error))
return
- # A failed push is logged and captured inside `push_registry`, never raised.
await asyncio.to_thread(push_inventory, inventories)
@@ -532,6 +561,26 @@ def _should_continue_as_new() -> bool:
return workflow.info().is_continue_as_new_suggested()
+def _finished_tick(result: OrchestrateResult, cutoff_iso: str) -> OrchestrateResult:
+ """Records how long the tick took from its scheduled minute.
+
+ The tick logs no other line, so this one is how an investigation reads a single tick's numbers.
+ """
+ duration = workflow.now() - dt.datetime.fromisoformat(cutoff_iso)
+ safe_record(record_tick_end, duration=duration, deadline_reached=result.deadline_reached)
+ # The log formatter drops `extra` fields, so the numbers go in the message.
+ workflow.logger.info(
+ "Alerts platform tick finished: scheduled_at=%s duration_ms=%d dispatched=%d remaining=%d pages=%d deadline_reached=%s",
+ cutoff_iso,
+ int(duration.total_seconds() * 1000),
+ sum(page.dispatched for page in result.pages),
+ result.remaining,
+ len(result.pages),
+ result.deadline_reached,
+ )
+ return result
+
+
@workflow.defn(name="alerts-platform-orchestrate")
class AlertsPlatformOrchestrateWorkflow(PostHogWorkflow):
"""One minute tick. Discovers demand once, then pages source dispatchers until the demand is
@@ -550,6 +599,12 @@ class AlertsPlatformOrchestrateWorkflow(PostHogWorkflow):
SearchAttributeKey.for_datetime("TemporalScheduledStartTime"), info.workflow_start_time
)
now = workflow.now()
+ record_schedule_to_start_latency(
+ "alerts_platform_tick_start_delay_ms",
+ scheduled_time=cutoff,
+ started_time=info.start_time,
+ description="Time from a tick's scheduled minute to its first workflow task",
+ )
hard_stop = info.execution_timeout or (TICK_DISPATCH_BUDGET + TICK_HARD_STOP_MARGIN)
inputs = replace(
inputs,
@@ -572,6 +627,7 @@ class AlertsPlatformOrchestrateWorkflow(PostHogWorkflow):
)
demand = discovered.batch_keys_by_source
inputs = replace(inputs, omitted=sum(discovered.omitted_by_source.values()))
+ safe_record(record_tick_keys, sum(len(keys) for keys in demand.values()) + inputs.omitted)
else:
demand = inputs.demand
@@ -589,7 +645,9 @@ class AlertsPlatformOrchestrateWorkflow(PostHogWorkflow):
+ inputs.omitted
+ sum(recorded.undispatched for recorded in pages)
)
- return OrchestrateResult(pages=pages, remaining=remaining, deadline_reached=True)
+ return _finished_tick(
+ OrchestrateResult(pages=pages, remaining=remaining, deadline_reached=True), cutoff_iso
+ )
# One list, so the settled results below can be read back against the source that
# produced each one.
paged = sorted(demand.items())
@@ -650,10 +708,13 @@ class AlertsPlatformOrchestrateWorkflow(PostHogWorkflow):
if demand and _should_continue_as_new():
workflow.continue_as_new(replace(inputs, page=page, demand=demand, pages=pages))
- return OrchestrateResult(
- pages=pages,
- remaining=inputs.omitted + sum(recorded.undispatched for recorded in pages),
- deadline_reached=False,
+ return _finished_tick(
+ OrchestrateResult(
+ pages=pages,
+ remaining=inputs.omitted + sum(recorded.undispatched for recorded in pages),
+ deadline_reached=False,
+ ),
+ cutoff_iso,
)
diff --git a/products/alerts_platform/backend/tests/test_inventory.py b/products/alerts_platform/backend/tests/test_inventory.py
index d7d7e85391b5..2d6e3387050c 100644
--- a/products/alerts_platform/backend/tests/test_inventory.py
+++ b/products/alerts_platform/backend/tests/test_inventory.py
@@ -18,7 +18,11 @@ from products.alerts_platform.backend.logic.inventory import (
count_inventory,
)
from products.alerts_platform.backend.models import PlatformAlert, PlatformAlertConfiguration
-from products.alerts_platform.backend.temporal.inventory_metrics import push_inventory
+from products.alerts_platform.backend.temporal.inventory_metrics import (
+ TickScheduleRuns,
+ push_inventory,
+ push_tick_schedule_runs,
+)
NOW = datetime(2026, 10, 23, 12, tzinfo=UTC)
LATER = NOW + timedelta(hours=1)
@@ -139,3 +143,15 @@ def test_push_replaces_the_group_with_the_dashboard_labels() -> None:
assert (
registry.get_sample_value("alerts_platform_alerts", {"source": "logs", "state": "firing", "muted": "true"}) == 2
)
+
+
+def test_schedule_runs_push_to_a_group_of_their_own() -> None:
+ with patch("posthog.metrics.push_registry") as push:
+ push_tick_schedule_runs(TickScheduleRuns(started=120, skipped_overlap=1, missed_catchup=3))
+
+ assert push.call_args.args[0] == "alerts_platform_tick_schedule"
+ registry = push.call_args.args[1]
+ assert {
+ outcome: registry.get_sample_value("alerts_platform_tick_schedule_actions", {"outcome": outcome})
+ for outcome in ("started", "skipped_overlap", "missed_catchup")
+ } == {"started": 120, "skipped_overlap": 1, "missed_catchup": 3}
diff --git a/products/alerts_platform/backend/tests/test_temporal_workflows.py b/products/alerts_platform/backend/tests/test_temporal_workflows.py
index e2138a1888f0..d3e58846371d 100644
--- a/products/alerts_platform/backend/tests/test_temporal_workflows.py
+++ b/products/alerts_platform/backend/tests/test_temporal_workflows.py
@@ -663,6 +663,7 @@ async def test_discovery_uses_scheduled_cutoff_or_manual_start(scheduled: bool)
)
info = MagicMock(
workflow_start_time=actual_start,
+ start_time=actual_start,
typed_search_attributes=attributes,
workflow_id="tick",
run_id="run",
@@ -675,6 +676,7 @@ async def test_discovery_uses_scheduled_cutoff_or_manual_start(scheduled: bool)
workflow, "execute_activity", AsyncMock(return_value=AlertDemand(batch_keys_by_source={}))
) as discover,
patch.object(workflow, "start_child_workflow", AsyncMock()) as dispatch,
+ patch.object(workflow, "logger"),
):
result = await AlertsPlatformOrchestrateWorkflow().run(OrchestrateInputs())
assert discover.await_args is not None