Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

3.1 KB
Raw Blame

Experiment metric recalculation cancellation

Experiment recalculations share result rows keyed by (experiment, metric_uuid, query_to). Stopped experiments reuse their fixed end date, so an abandoned run and its replacement can write the same row.

Result persistence locks the experiment and then the recalculation row in one transaction. The status check and result upsert share that transaction, so a terminal transition either follows the old write or prevents it. Missing, completed, and failed recalculations cannot write results. The lock order matches request_recalculation and avoids a deadlock between supersession and the experiment foreign-key check on a new result. ClickHouse queries and statistics run outside these locks.

The activity shields its synchronous calculation because canceling an asyncio task cannot stop a running worker thread. After local activity cancellation, it waits for the calculation with a cleanup deadline of METRIC_CALC_ACTIVITY_TIMEOUT_SECONDS from the first cancellation. Repeated cancellation does not restart or interrupt that wait. The activity propagates the original cancellation even if the calculation fails during cleanup.

This cleanup preserves the worker's activity slot while it waits; it does not stop the query or immediately release its memory. The deadline bounds cleanup, not the underlying thread's lifetime. Precomputation, executor queuing, statistics, and database waits can extend beyond the final ClickHouse query's execution limit. A calculation can therefore continue after the cleanup deadline expires.

Temporal delivers a workflow cancellation to a running activity only in a heartbeat response. The calc activity heartbeats every METRIC_CALC_HEARTBEAT_TIMEOUT_SECONDS / 4 while the calculation thread runs, and the workflow sets that heartbeat timeout on the call. A workflow cancellation, or a timeout the server enforces, therefore reaches the activity and starts the cleanup above, which kills the active ClickHouse query. Precomputation and the query runner do not cancel cooperatively, so the thread can still run past the kill. Result-write protection must remain independent of cancellation delivery.

See the workflow architecture and Temporal's cancellation contract.

Superseded timeseries backfills

A request with the same metric and calculation key reuses the active backfill. A different key cancels the old request and starts a replacement. Each daily write locks the experiment and checks the request status in the same transaction, preventing a superseded worker from overwriting the replacement's days. Queries run outside the transaction.

The request stores the metric's current definition and key, and ignores a definition or key in the request body. Until backfills consume a frozen calculation config, each day's runner must give the request's key, in the key version of that key, before executing. Settings drift cancels the request instead of storing a result under settings it did not use.