Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(experiments): tag recalculation queries with the offline workload (#124492)

Rodrigo Iloroecb3960e0f20
diff --git a/products/experiments/backend/temporal/ARCHITECTURE.md b/products/experiments/backend/temporal/ARCHITECTURE.md
index e73ed0952a2b..33235cfb6b55 100644
--- a/products/experiments/backend/temporal/ARCHITECTURE.md
+++ b/products/experiments/backend/temporal/ARCHITECTURE.md
@@ -283,9 +283,9 @@ The split is intentional: Grafana tells you about the worker process, PostHog te
Temporal delivers a cancellation only in a heartbeat response, so a workflow cancel or a server-side timeout now reaches the running attempt and starts the cleanup above.
A dead worker misses its heartbeats, so Temporal retries the metric after 60s instead of after the full 300s attempt.
See [cancellation and result-write protection](../../../../docs/internal/experiment-metric-recalculation.md) for the locking contract and cleanup limits.
-- **Calc queries run on the offline ClickHouse nodes.** The calc activity builds its `ExperimentQueryRunner` with `workload=Workload.ONLINE`. That workload uses `CLICKHOUSE_HOST`, and the recalculation worker deployments set `CLICKHOUSE_HOST` to the offline host. So the queries reach the offline replicas, not the replicas that serve interactive product traffic. The activity task queue above caps how many of them run at once.
+- **Calc queries run on the offline ClickHouse nodes.** Recalculation runs only on the offline replicas by design; it has no online tier. The calc activity builds its `ExperimentQueryRunner` with `workload=Workload.OFFLINE`, so the query tag states where the queries run. The recalculation worker deployments set no `CLICKHOUSE_OFFLINE_CLUSTER_HOST`, so `OFFLINE` falls back to `CLICKHOUSE_HOST`, which those deployments set to the offline host. The activity task queue above caps how many of the queries run at once.
- The workload stays set explicitly rather than left to the default. The base `QueryRunner` default is `Workload.DEFAULT`, which follows the process default (Celery pins every task to offline process-wide, and API-key traffic is forced offline). Passing `Workload.ONLINE` keeps the routing the same however the activity is invoked, and the worker's host setting decides where it goes.
+ The workload stays set explicitly rather than left to `Workload.DEFAULT`, which follows the process default and changes with how the activity is invoked.
Without the flag below, the queries run as the `default` user. The calc tags them with the cache-warming feature, so `sync_execute` asks for the `cache_warmup` user. The recalculation workers do not wire that user, so `get_clickhouse_creds` falls back to the pod's `default` user, and the queries count against its per-node limit on concurrent queries, which other products share.
diff --git a/products/experiments/backend/temporal/recalculation_logic.py b/products/experiments/backend/temporal/recalculation_logic.py
index 74a397473fc3..bc3bbdd3eabc 100644
--- a/products/experiments/backend/temporal/recalculation_logic.py
+++ b/products/experiments/backend/temporal/recalculation_logic.py
@@ -736,7 +736,7 @@ def _calculate_experiment_metric_for_recalculation_sync(
query=ExperimentQuery(experiment_id=experiment_id, metric=build_metric(metric_dict)),
team=experiment.team,
as_of=query_to_dt,
- workload=Workload.ONLINE,
+ workload=Workload.OFFLINE,
# Userless background recompute. Warehouse access is enforced when the metric is authored,
# so resolve warehouse tables here instead of failing closed.
bypass_warehouse_access_control=True,
diff --git a/products/experiments/backend/temporal/test_recalculation_activities.py b/products/experiments/backend/temporal/test_recalculation_activities.py
index 464327178fdb..cd419ecf1b4f 100644
--- a/products/experiments/backend/temporal/test_recalculation_activities.py
+++ b/products/experiments/backend/temporal/test_recalculation_activities.py
@@ -21,7 +21,7 @@ from temporalio.exceptions import ApplicationError
from temporalio.testing import ActivityEnvironment
from posthog.clickhouse.client import sync_execute
-from posthog.clickhouse.client.connection import ClickHouseUser
+from posthog.clickhouse.client.connection import ClickHouseUser, Workload
from posthog.clickhouse.client.limit import ConcurrencyLimitExceeded
from posthog.clickhouse.query_tagging import Feature, Product, get_query_tags, tags_context
from posthog.exceptions import ClickHouseAtCapacity, ClickHouseQueryMemoryLimitExceeded, ClickHouseQueryTimeOut
@@ -1079,6 +1079,7 @@ class TestCalculateActivity(BaseTest):
mock_runner.assert_called_once()
kwargs = mock_runner.call_args.kwargs
assert kwargs["as_of"] == datetime.fromisoformat(_QUERY_TO)
+ assert kwargs["workload"] == Workload.OFFLINE
def test_query_from_matches_experiment_start_date_on_result_row(self):
# Companion to test_query_to_is_passed_as_as_of_to_runner: the lower bound of the run's