:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Web analytics precomputation
PostHog web analytics has two production precomputation systems, plus an experimental third (the dimensional tables). They target the same problem (avoid scanning raw events on every dashboard load) but use different mechanisms and apply to different query shapes.
For how these tiers fit into the full serving ladder (fast paths, full join, dispatch order, query_type tags), see docs/internal/web-analytics-query-serving.md.
The systems
v2 pre-aggregated tables
Status: deprecated. No new enrollments — retained only for the largest existing customers until lazy computation fully replaces it, at which point this system goes away.
DAG-warmed ClickHouse tables (web_pre_aggregated_stats, web_pre_aggregated_bounces) that store hourly-rollup data computed in the background. Gated per-team by the useWebAnalyticsPreAggregatedTables modifier plus the SETTINGS_WEB_ANALYTICS_PRE_AGGREGATED_TABLES feature flag.
- Owned by: web analytics team
- Population: scheduled Dagster jobs in
products/web_analytics/dags/ - Coverage: stats table queries, web overview (partial)
- Adoption (as of 2026-05): effectively zero — only one team has the modifier on in prod
Dimensional pre-aggregated tables (experimental)
Status: experimental, write-only. The precomputation-framework successor to the v2 tables: an experiment in whether v2's fixed-dimension read model can run on the lazy-computation framework (per-team jobs + TTLs) instead of v2's staging-table + partition-swap ETL. No read path is merged — the runner never reads these tables today (PR #62437, the read path, is closed).
Two ClickHouse tables carry the same fixed dimension set as v2 (host, device_type, pathname, browser, os, viewport, referring_domain, utm_*, geoip, …):
web_stats_dimensional_preaggregated— stats (persons / sessions / pageviews states).web_bounces_dimensional_preaggregated— bounces (adds bounce / duration / session-count states).
Both are ReplacingMergeTree (version column computed_at), sharded by sipHash64(job_id) on the AUX cluster, partitioned by toYYYYMMDD(expires_at).
- Owned by: web analytics team, on the analytics_platform framework.
- Population: the
web_dimensional_precompute_jobDagster job, every 20 minutes (products/web_analytics/dags/web_dimensional_precompute.py). For each team on a region-scoped allowlist it drivesensure_precomputedover a rolling 92-day window, chunked to one UTC day per INSERT. 92 UTC days hold the start of a-90dread in every team timezone. Every window is aPreaggregationJob(Postgres) with ajob_idand a TTL by age (today 45 minutes, ≤2 days 1 day, older 90 days plus a stable per-day offset of up to 7 days); re-runs recompute only expired windows. The offset spreads older days over different runs, so each run rebuilds a few of them. Each run builds both tables for a day, newest day first. The TTL bands start at UTC midnight, like the windows, so the current UTC day keeps its 1h TTL in every team timezone. The allowlist defaults per Cloud region (DEFAULT_ROLLOUT_TEAM_IDS_BY_REGION) and is overridable viaWEB_DIMENSIONAL_PRECOMPUTE_TEAM_IDS; an empty list disables the job. The inserts carry the legacy job's memory and time ceilings, because one UTC day of a very large team outgrew the interactive defaults (#117339). They also narrow HogQL's 3-day session lookback to 25 hours, after the wider read ran the shared sessions node out of memory (#117872). The job has no stale grace or forced refresh of its own: today's window rebuilds only when a run finds its job expired. So today's data can lag by up to the TTL plus one cron period (20 minutes). - Monitoring: after the writers, a second op reads the READY jobs back for each team and table, merges their intervals, and fails the run when a UTC day in the writers' window has no fresh job or the newest job for the writers' current day is older than the TTL plus two runs. The check takes the window and the current day from the writers, so a run that crosses UTC midnight does not report the new day as missing.
A failed run alerts
#alerts-web-analyticsthroughnotify_slack_on_failure, which routes on the job'sownertag. The op also pushesweb_dimensional_precompute_coverage_missing_daysandweb_dimensional_precompute_today_lag_secondsgauges, labeled byteam_idandtable, to the Prometheus push gateway. - Coverage: the same stats-breakdown and overview shapes v2 serves — about 45% of user-facing web-analytics reads by shape (stats breakdowns on the stored dimensions, plus overview). Not goals, vitals, frustration metrics, external clicks, or the other engines (Trends / Retention / HogQL). Its differentiator over lazy: arbitrary multi-dimension filters within the fixed set resolve from one table, with no per-filter-combination warming.
- Adoption: none in prod — a write-only pilot for a v2 side-by-side.
Read-cost finding (2026-08): for clean single-generation reads these tables match v2 on rows and bytes (within ±7%) and use far less query memory.
Because job_id is in the ORDER BY, recompute generations never collapse, so a read spanning un-deduped generations inflates scan cost roughly linearly with the generation count — that, not the schema, is the historical "not as fast" result.
Reviving the read path needs precise ready-job filtering plus a way to collapse superseded generations.
Schema: posthog/clickhouse/preaggregation/web_*_dimensional_preaggregated_sql.py; INSERT templates: products/web_analytics/backend/hogql_queries/web_dimensional_precompute.py.
Lazy computation
The newer general-purpose framework at products/analytics_platform/backend/lazy_computation/. Builds precomputed buckets in the background, caches them in a dedicated CH table per query family, and serves reads from the cache; a user read that misses serves the live path and enqueues a debounced background warm. Gated per-org by the web-analytics-precompute-toggle PostHog feature flag (evaluated against the team's organization).
- Owned by: web analytics team, riding on the analytics_platform framework
- Population: background-only since #72959 — user reads never build inline; a miss serves live and enqueues a debounced background warm. The hourly demand-driven warmer (
dags/cache_warming.py) and stale revalidation keep hot shapes fresh - Coverage (today): overview, goals, vitals path breakdown, and three stats-table families — paths (
WebStatsBreakdown.PAGE/INITIAL_PAGE+includeBounceRate), frustration metrics, and simple breakdowns. See the*_lazy_precompute.pymodules inproducts/web_analytics/backend/hogql_queries/ - Adoption (as of 2026-05): freshly enabled, org feature flag gates further rollout
When to use which
Both systems can coexist. The runner tries each in order; if both miss or are disabled the runner falls through to a raw events scan.
| You want… | Use |
|---|---|
| A new query family where the cache shape is bounded and stable | lazy computation |
| Coverage of an existing query family that v2 already handles | v2 if a team already has v2 enabled; otherwise lazy |
| Per-team custom precompute logic (uncommon) | lazy computation |
| Background warming on a schedule | either — lazy has the hourly eager + demand warmers; v2 uses its own Dagster DAGs |
A miss never builds inline on either system: since #72959 run_inserts is true only for background-warming requests, so the first read of a cold shape serves the live path while the warm happens behind it.
Lazy computation for web overview (current production path)
WebOverviewQueryRunner._calculate runs three checks in order:
- Lazy precompute (
can_use_lazy_precompute+execute_lazy_precomputed_read) — the path described below. - v2 pre-agg (
get_pre_aggregated_response) — the legacy path; falls through unless the team modifier is on. - Raw events scan — the original path.
Schema
web_overview_preaggregated (sharded by sipHash64(job_id), partitioned by toYYYYMMDD(expires_at), ReplicatedReplacingMergeTree with computed_at as the version column). Five aggregate state columns matching the runner's metric tuple:
uniq_users_state AggregateFunction(uniq, UUID)— unique persons (HLL ~99%)uniq_sessions_state AggregateFunction(uniq, String)— unique sessions (HLL ~99%)sum_pageviews_state AggregateFunction(sum, Int64)— pageviews/screensavg_duration_state AggregateFunction(avg, Float64)— average session duration (seconds)avg_bounce_state AggregateFunction(avg, Int64)— bounce rate (0/1 per session)
Partition key on expires_at rather than time_window_start so ttl_only_drop_parts=1 can drop expired parts atomically.
Bucketing and timezones
Buckets are UTC hourly. The read converts the team's local request window to UTC and filters bucket boundaries on hour edges. This means:
- Whole-hour-offset timezones (UTC, PT, ET, JST, etc. — ~99% of teams) read exact-to-the-hour data.
- Half-hour-offset timezones (IST
+5:30, Newfoundland-3:30, Nepal+5:45, Iran+3:30) cannot be served correctly by UTC hourly buckets — thecan_use_lazy_precomputegate refuses them viais_integer_timezone(). They fall through to v2/raw.
Sessions that straddle bucket boundaries
The framework chunks the precompute span into daily UTC jobs. Each job's INSERT scans [time_window_min, time_window_max) for events, groups by session_id, and emits one hourly bucket row per session, keyed on toStartOfHour(min(session.$start_timestamp)).
A session that starts at 23:30 UTC with an event at 00:15 UTC the next day spans two daily jobs. Without help, the first job would only see the 23:30 event and miscount that session's pageviews.
To fix this, the INSERT widens the event scan forward by SESSION_FORWARD_PAD_MINUTES (currently 24 h) past the job's [time_window_min, time_window_max) window. The HAVING clause keeps each session attributed to its actual start hour — over-scan adds INSERT cost but cannot produce duplicate rows in any bucket. Forward-only is sufficient because the HAVING keeps only sessions whose min(session.$start_timestamp) falls inside the window; every event of such a session has timestamp >= time_window_min, so backward scanning never picks anything that survives HAVING.
The 24 h pad matches the JS SDK's hard SESSION_LENGTH_LIMIT and covers effectively 100% of population sessions (measured p99 ≈ 79 min, with a long tail). Sessions exceeding 24 h are documented as undercounted on cross-boundary days. The long-term fix is to drive the INSERT from raw_sessions (bounded by the embedded UUIDv7 timestamp), which removes the pad entirely.
Eligibility gate
can_use_lazy_precompute(runner) in web_overview_lazy_precompute.py. Refuses when any of:
- The
web-analytics-precompute-togglePostHog feature flag is off for the team's organization - Team timezone has a non-whole-hour UTC offset
query.conversionGoalis setquery.sampling.enabledis Truequery.modifiers.sessionsV2JoinMode == "uuid"(column type mismatch — temporary; should be re-enabled by re-typinguniq_sessions_stateto(uniq, UUID))- The team has any property-level access controls (
team_has_property_access_rules) — precompute results are built userless and shared by a user-independent cache key, so they can't honor per-user property restrictions; the query falls through to the live path, which enforces them per requesting user - Date range exceeds
MAX_PRECOMPUTE_DAYS - Either date_from or date_to is None
Any event/person filter shape is accepted: the query.properties list is translated via property_to_expr, and each distinct filter set becomes its own cache key (namespace). Session and cohort filters are refused (they fall through to the live path) — the userless precompute INSERT would apply them, but the live runners handle those types differently per family (web vitals drops them), so precomputing them would serve a different population than the live fallback. Filters the INSERT can't express fail the job and fall back to the live query automatically. Because any filter combination mints a new namespace, web_ensure_precomputed enforces a per-team distinct-shape ceiling (WEB_ANALYTICS_PRECOMPUTE_MAX_SHAPES_PER_TEAM, default 1000): once a team holds that many live shapes, a build for a new shape drops to a check-only pass and serves live instead, while shapes the team already holds keep serving and refreshing. It is a coarse backstop, not a quota; 0 disables it.
When the gate returns False the runner silently falls through to v2 / raw. Today there is no telemetry on gate rejections — operators tuning the rollout have to read the source to know why a team isn't seeing the lazy path. A web_overview_lazy_gate_rejected_total{reason} counter would close that gap (open issue).
TTL schedule
Different freshness per how recent the data is:
| Day age | TTL | Rationale |
|---|---|---|
| Today | 4 h | Must outlast the hourly eager warmer so user reads never race a refresh |
| Yesterday | 6 h | Distinct from today's TTL so split_ranges_by_ttl keeps the two days separate |
| 2–7d | 5 days | Session-final (24h session cap) — recomputing an immutable window buys nothing |
| 8–14d | 7 days | Distinct per-week TTLs force weekly job boundaries, keeping INSERT scans bounded |
| 15–21d | 10 days | — |
| 22–35d | 12–14 d | — |
| 36d+ | 90 days | A year-long shape must not rescan a year of events each time the band expires |
Stored via the LAZY_TTL_SECONDS dict in web_lazy_precompute_common.py (see the comment there for the full freshness-vs-job-sizing reasoning); consumed by lazy_computation_executor.parse_ttl_schedule against the team's timezone.
For the framework mechanisms behind it, see design decisions.
A user forced refresh treats today's bucket as expired for that read only (#86699): the 4-hour today band otherwise left hourly graphs hours behind after a Reload.
The forced read goes live, but the stored bucket is not rebuilt early: the revalidation it enqueues runs as a background build with the normal TTL, finds the job still fresh, and reuses it.
Later non-forced reads keep serving that bucket until its normal TTL lapses.
Background builds never get this rule, because the same schedule stamps expires_at and the revalidation task runs in the forced mode, so it would store buckets that expire after 1 second.
A forced refresh also never gets the stale grace, or repeated Reloads kept serving the stale row they were meant to clear (#73693).
Read path
The read is a single sync_execute call (not HogQL — see "Why bypass HogQL" below). It computes the 5 metric pairs (current + previous period) via *MergeIf aggregates filtered by the team-tz date range converted to UTC. Settings:
load_balancing="in_order"— paired with the INSERT side's same setting for read-your-writes via Approach E in CONSISTENCY.md.optimize_skip_unused_shards=1—job_id IN (...)+ sharding-by-sipHash64(job_id)lets ClickHouse prune to the right shards.
Result is built into the standard WebOverviewQueryResponse via _build_response_from_row, with preComputeStrategy=WebAnalyticsPreComputeStrategy.LAZY_PRECOMPUTE.
Why bypass HogQL
HogQL HogQLGlobalSettings is extra="forbid", so we couldn't get arbitrary CH settings (originally select_sequential_consistency=1, since dropped) through execute_hogql_query. The read query shape is stable enough that sync_execute with parameterized values is appropriate. The team_id WHERE clause is enforced manually (the HogQL printer's auto-injected team_id_guard_for_table is not applied for sync_execute).
If we later move back to HogQL (after the consistency story is settled), the HogQL table registration at posthog/hogql/database/schema/web_overview_preaggregated.py is still present and ready.
Observability
- Read query: tagged
query_type="web_overview_lazy_query"insystem.query_log.log_comment. - INSERT query: tagged
query_type="web_overview_lazy_insert"(via the framework'squery_typekwarg added in this commit family). - Failures:
web_overview_lazy_precompute_failed_total{error_type}Prometheus counter (bounded by Python exception class). - Cache warmer: the DAG at
products/web_analytics/dags/cache_warming.pyrecognizes bothweb_overview_queryandweb_overview_lazy_query. - Adoption / latency: see
evaluating-web-analytics-performanceor querysystem.query_logdirectly.
Known limitations / open issues
- No gate-rejection telemetry — operators can't see why a team isn't seeing the lazy path. Counter pending.
error_typePrometheus label is too coarse —ServerExceptionfrom CH could mean quorum / memory / schema. Should bucket by CH error code.- UUID session mode rejected, not handled. Re-typing
uniq_sessions_stateto(uniq, UUID)would lift this; tracked as a follow-up. - Empty
*MergeIfrows are treated as legitimate empty windows. Emptysync_executeresult on this query shape is almost always a driver/transport error rather than "no data"; should be treated as failure + fall through. - HogQL
top_level_settingsonWebOverviewPreaggregatedTableis currently unused (read bypasses HogQL); kept as a hook if we re-route through HogQL later.
Adding lazy computation to another web analytics query family
Reference implementation: products/web_analytics/backend/hogql_queries/web_overview_lazy_precompute.py.
Roughly:
- Schema — new CH table under
posthog/clickhouse/preaggregation/. ReplacingMergeTree with(team_id, job_id, time_window_start)ORDER BY, sharded bysipHash64(job_id), partitioned bytoYYYYMMDD(expires_at). Register inposthog/clickhouse/schema.pytest fixtures. - Migration —
posthog/clickhouse/migrations/0XXX_<name>.py. Sharded + distributed CREATE onNodeRole.DATA. - HogQL table —
posthog/hogql/database/schema/<name>.py. Register inposthog/hogql/database/database.py. LazyComputationTableenum — add a value inproducts/analytics_platform/backend/lazy_computation/lazy_computation_executor.py.- Insert query — HogQL template using
{time_window_min}/{time_window_max}placeholders. Add a session-boundary pad if the query has session-level joins. - Read —
sync_executewith parameterized SQL, settings ={"load_balancing": "in_order", "optimize_skip_unused_shards": 1}. Filter byteam_idANDjob_id IN (...)in the WHERE clause. - Runner integration — add a
can_use_*gate and anexecute_*_readorchestrator; short-circuit in_calculatebefore the v2/raw fallthrough. - Tests — round-trip (lazy == raw) parameterized over team timezones, gate fallthrough for each disqualifying condition, half-hour-offset fallthrough, cache hit (second call doesn't create new jobs).
- Cache warmer — add the new
query_typeto the warmer DAG's allowlist inproducts/web_analytics/dags/cache_warming.py.
Rollout order across query families: web_overview_query (shipped), stats_table_path_bounce_query (shipped — this PR), stats_table_main_query (next), then web_goals_query deferred for custom-goal-definition complexity.
Lazy computation for the PATHS tile
WebStatsTableQueryRunner._calculate adds a fourth check: lazy precompute first for the WebStatsBreakdown.PAGE + includeBounceRate combination only. Other breakdowns and other column combinations fall through to the existing v2/raw paths.
Schema
web_stats_paths_preaggregated (sharded by sipHash64(job_id), partitioned by toYYYYMMDD(expires_at), ReplacingMergeTree with computed_at as the version column). One row per (team_id, job_id, time_window_start, breakdown_value):
breakdown_value String— pathname, optionally prefixed with$hostwhen the query hasincludeHost.uniq_users_state AggregateFunction(uniq, UUID)— persons that touched this pathname.sum_pageviews_state AggregateFunction(sum, Int64)— pageview/screen events on this pathname.avg_bounce_state AggregateFunction(avg, Nullable(Float64))—if(pathname == entry_pathname, is_bounce, NULL)averaged across rows.avg's null-skip semantics make this equivalent to "bounce rate of sessions that entered on this pathname" — matching the v2PATH_BOUNCE_QUERYjoin semantic without a JOIN at read time.
The state is Nullable(Float64) so the if(..., NULL) expression in the INSERT round-trips into the column without an explicit toNullable coercion.
Eligibility gate
can_use_lazy_precompute(runner) in web_stats_paths_lazy_precompute.py. Shares the common gate (web_lazy_precompute_common.py) with web overview — same org flag, same timezone / sampling / UUID-mode / filter rules. Adds PATHS-specific refusals:
query.breakdownBy != WebStatsBreakdown.PAGEquery.includeBounceRateis False (the lazy table is purpose-built for bounce-augmented paths)query.includeAvgTimeOnPageis True (not yet wired)query.includeScrollDepthis True (not yet wired)
Read path
Single sync_execute over web_stats_paths_preaggregated with uniqMergeIf / sumMergeIf / avgMergeIf covering both current and previous periods. The runner builds the standard WebStatsTableQueryResponse (breakdown_value + visitor/views/bounce-rate tuples + ui_fill_fraction + cross_sell). Sorting, paging, and fill-fraction are computed in Python over the materialized result set, matching PathBounceStrategy defaults (visitors DESC, then breakdown_value ASC; WebAnalyticsOrderByFields overrides honored for VISITORS / VIEWS / BOUNCE_RATE).
Known follow-ups
- INITIAL_PAGE + bounce (entry-pathname tab) is a different SQL shape — separate precompute table or shared one with an entry-only state column.
- The response's
preComputeStrategyis set toWebAnalyticsPreComputeStrategy.LAZY_PRECOMPUTE; the frontend'sPreAggregatedBadgekeys off it (a distinct "precomputed" variant) so the lazy path is visually distinguishable from the v2PRE_AGGREGATEDpath.
Lazy computation for the web vitals path-breakdown tile
WebVitalsPathBreakdownQueryRunner._calculate follows the same gate-then-fallthrough shape as web overview:
- Lazy precompute (
can_use_lazy_precompute+execute_lazy_precomputed_read) — short-circuits when eligible and returns immediately. - Raw events scan — the original
quantile(p)(toFloat(properties.$web_vitals_*_value))per path.
Schema
web_vitals_paths_preaggregated (sharded by sipHash64(job_id), partitioned by toYYYYMMDD(expires_at), ReplicatedReplacingMergeTree with computed_at as the version column). One row per (team, job, hour, path), four state columns — one per Web Vitals metric:
inp_quantiles_state AggregateFunction(quantiles(0.75, 0.90, 0.99), Float64)lcp_quantiles_state AggregateFunction(quantiles(0.75, 0.90, 0.99), Float64)cls_quantiles_state AggregateFunction(quantiles(0.75, 0.90, 0.99), Float64)fcp_quantiles_state AggregateFunction(quantiles(0.75, 0.90, 0.99), Float64)
Each state holds one reservoir covering all three percentiles. Reads pick the queried percentile via arrayElement(quantilesMergeIf(0.75, 0.90, 0.99)(state, range_filter), pct_index). Same reservoir algorithm as the raw quantile(p) — exact when unsaturated, within sampling noise once it is.
Four columns vs. a metric discriminator: ARRAY JOIN would fan one event into four rows, but the new ClickHouse analyzer rejects bare events.properties references inside the ARRAY JOIN source array (the source array is resolved before the FROM alias scope). Four columns let the INSERT stay a single FROM events GROUP BY (hour, path), no fan-out, and each metric tab reads exactly one column.
Bucketing and timezones
Daily, team-tz aligned. Bucket key is toStartOfDay(timestamp, team_tz) — start of the team's local day. The underlying Unix timestamp stored in time_window_start is the UTC instant of that local midnight, so reads filter against the team-tz date range converted to UTC and get exact alignment.
This differs from web overview / web stats which use UTC-hourly buckets:
- The path-breakdown tile only consumes day-aligned date ranges from the dashboard filter, so a daily bucket is sufficient and ~24× smaller than hourly.
- Bucketing in the team's tz means half-hour-offset timezones (IST +5:30, Newfoundland -3:30, Nepal +5:45, Iran +3:30) are supported too — this runner opts out of the shared
is_integer_timezonegate. - A UTC-daily INSERT job typically writes into TWO team-tz day buckets (events in the first hours of UTC day N belong to team-tz day N-1 for non-UTC teams). The
ReplacingMergeTreekey(team_id, job_id, time_window_start, path)keeps the rows distinct per job; reads merge them viaquantilesMergeIffor full team-tz day coverage.
No session join in the raw query, so no SESSION_FORWARD_PAD_MINUTES — each event maps to exactly one (team-tz day, path) bucket.
Read
Mirrors the raw query's outer shape:
SELECT multiIf(value <= good, 'good', value <= needs_improvements, 'needs_improvements', 'poor') AS band, path, value
FROM (
SELECT path,
arrayElement(quantilesMergeIf(0.75, 0.90, 0.99)(<metric>_quantiles_state, time_filter), pct_index) AS value
FROM posthog.web_vitals_paths_preaggregated
WHERE team_id = ? AND job_id IN (...)
GROUP BY path HAVING value >= 0
)
ORDER BY value ASC, path ASC
LIMIT 20 BY band
The runner re-partitions the resulting (band, path, value) tuples into the good / needs_improvements / poor arrays the response expects.
Eligibility gate
can_use_lazy_precompute in products/web_analytics/backend/hogql_queries/web_vitals_paths_lazy_precompute.py delegates to the shared gate with require_integer_timezone=False (see "Bucketing and timezones" above). The shared gate rejects: org feature flag off, explicit per-query opt-out (useWebAnalyticsPrecompute=False — an untouched toggle defaults on), conversion goal, sampling enabled, sessionsV2JoinMode=uuid, missing date range, and date range over MAX_PRECOMPUTE_DAYS. Any event/person property-filter shape is accepted (each distinct set becomes its own cache key, bounded by the per-team shape ceiling — see the web overview eligibility gate above); session/cohort filters fall through to live.
Observability
- Read query: tagged
query_type="web_vitals_paths_lazy_query". - INSERT query: tagged
query_type="web_vitals_paths_lazy_insert". - Failures:
web_vitals_paths_lazy_precompute_failed_total{error_type}Prometheus counter. - Cache warmer:
web_vitals_paths_lazy_queryis in the warmer DAG allowlist inproducts/web_analytics/dags/cache_warming.py.
Known limitations
WebVitalsQuery(line-chart tile) is not covered. That query wraps aTrendsQueryand dispatches throughTrendsQueryRunner; lazy precompute for it would need a different shape and is deferred.- Adding a metric (e.g. TTFB) is a schema change — add the column to
web_vitals_paths_preaggregated, the HogQL table registration, the INSERT template, and the_METRIC_STATE_COLUMNmap. - Bands are computed in ClickHouse from the runtime thresholds, not stored — so a threshold change is free on the read side.
Eager baseline warming (hourly Dagster job)
A cold or expired shape's first read serves the live path and only warms in the background — but for high-traffic teams the dashboard's main tiles are requested constantly, and there's no reason to let the first user of every cycle take that live-path miss. The eager job pre-warms the same lazy precompute cache (and the Django response cache) for a fixed query matrix, ahead of users.
- Location:
products/web_analytics/dags/eager_web_analytics_precompute.py - Schedule:
53 * * * *(hourly, with its start outside the demand warmer's release window); skipped if a prior run is still in flight (check_for_concurrent_runs). - Window: trailing 28 days. The lazy precompute stores per-day buckets, so a 28-day warm naturally covers any sub-window the dashboard asks for.
- Matrix per team:
WebOverviewQuery+WebGoalsQuery+WebVitalsPathBreakdownQuery+ oneWebStatsTableQueryperWebStatsBreakdownrendered by the dashboard (~23 breakdowns includingFrustrationMetrics). - Per-query toggle: every warmer query sets
useWebAnalyticsPrecompute=Trueexplicitly. Precompute now defaults on for enrolled teams (only an explicitFalseopts out), so this is redundant — kept to make the warmer's intent explicit. - Freshness handoff: each payload is dispatched via
get_query_runner(...).run(...). The runner routes through its family's*_lazy_precompute.pymodule, which callsensure_*_precomputed— already idempotent. The DAG does not enumerate windows or inspect job state; the runner is the source of truth for what's stale. - Audience: teams belonging to organizations rolled out on the
web-analytics-precompute-togglefeature flag — the same flag the runtime lazy read path checks. The job parses the flag'sMatch organizations against id equals <uuid>group conditions and resolves them to teams viaTeam.objects.filter(organization_id__in=...). The flag lives on PostHog's internal dogfooding project; self-hosted instances are gated out viais_cloud()so a same-keyed flag on someone else's team-2 doesn't trigger anything. - Audience cap: 200 teams. A typo in the flag config fails-loudly (op returns with
skipped=Nand zero warmed) rather than silently overloading ClickHouse. - Cycle budget: 45 minutes of wall-clock; remaining teams are reported as
skippedif the budget is exhausted. The concurrency guard absorbs the next tick.
Because the eager job and the lazy read path consult the same flag, the warming audience never drifts from the audience the read path will actually serve — there is no second flag to keep in sync.
This job is complementary to cache_warming.py, which replays whatever queries users actually ran in the last N days. The eager job covers the fixed UI matrix; the replay job covers the long tail of team-specific filter combinations.
Trend tiles and Active Hours (precompute-backed trends)
The dashboard's graph tiles run vanilla TrendsQuery, routed at dispatch to WebTrendsQueryRunner when the query carries tags.productKey == "web_analytics" AND the web-analytics-trends-precompute flag (locally evaluated, fails closed) is on for the team. The runner serves eligible single-series shapes — unique visitors, page views, sessions, avg session duration, bounce rate — from the same hourly web_overview_preaggregated buckets the overview tile keeps warm (the query is mapped to an inner WebOverviewQuery, so the precompute job hash is identical and buckets are shared), and falls back to the standard trends path for everything else. The Active Hours unique-visitors tab is served the same way by WebCalendarHeatmapTrendsQueryRunner.
Rollout gates, in order:
WEB_ANALYTICS_TRENDS_PRECOMPUTE_TEAM_IDS(env, deploy-time) — dogfooding list that short-circuits the flag.web-analytics-trends-precomputePostHog feature flag — the percentage ramp; also folded into the runner's cache key so disabling it invalidates cached precompute-served results immediately.- The shared enrollment gate (
web-analytics-precompute-toggle/WEB_ANALYTICS_LAZY_PRECOMPUTE_TEAM_IDS) still applies underneath.
Accepted semantic differences vs vanilla trends (the tiles align with the overview tile instead): session-start-hour attribution, HLL uniques, $pageview + $screen counted together, sessionless events excluded. The per-query "Allow precompute" opt-out does not reach trend tiles (TrendsQuery deliberately carries no web-analytics fields); the flag is the kill switch.
The Active Hours path is stricter than the trend tiles: it falls back to the live heatmap for teams that track both $pageview and $screen (the buckets carry no event dimension, and the live heatmap filters exactly to the requested event), for teams aggregating by distinct ID, and for date bounds that are not hour-aligned in UTC (explicit sub-hour ranges, fractional-offset timezones).
Web Vitals tab timeseries
The Web Vitals tab's timeseries tile sends a WebVitalsQuery wrapper whose source is a TrendsQuery of four $web_vitals percentile series. With the web-analytics-vitals-precompute flag (locally evaluated, fails closed) on for the team, dispatch routes it to WebVitalsQueryRunner, which merges the per-path quantile states of the web_vitals_paths_preaggregated buckets into per-day tab-level percentiles — the same buckets the path-breakdown tile on the same tab keeps warm, so the inner ensure hashes to the sibling tile's job family and both tiles share one set of jobs. With the flag off the kind has no runner branch and process_query_model unwraps to the source TrendsQuery, the pre-existing live path.
Servable shapes: the canonical four-series tab query with a shared p75/p90/p99 percentile, day-aligned ranges, and day/week/month intervals (buckets are team-tz daily, so hour interval falls back). p95, compare, breakdowns, formulas, per-series filters, per-series math multipliers, and conversion goals fall back. Only the line-graph display is servable; any other display (total-value, cumulative, and the displays with dedicated runners — calendar heatmap, box plot, slope graph) unwraps to its source TrendsQuery so the source's own dispatch handles it. Accepted semantic difference vs the live path: events with a NULL $pathname are absent from the buckets, so they are excluded from the served percentiles.
Related code
products/web_analytics/backend/hogql_queries/web_overview.py— runnerproducts/web_analytics/backend/hogql_queries/web_overview_lazy_precompute.py— overview lazy pathproducts/web_analytics/backend/hogql_queries/web_overview_pre_aggregated.py— overview v2 pathproducts/web_analytics/backend/hogql_queries/stats_table.py— stats table runnerproducts/web_analytics/backend/hogql_queries/web_stats_paths_lazy_precompute.py— PATHS lazy pathproducts/web_analytics/backend/hogql_queries/web_lazy_precompute_common.py— shared eligibility gate + helpersposthog/clickhouse/preaggregation/web_overview_preaggregated_sql.py— overview schemaposthog/clickhouse/preaggregation/web_stats_paths_preaggregated_sql.py— PATHS schemaproducts/web_analytics/backend/hogql_queries/web_vitals_path_breakdown.py— vitals runnerproducts/web_analytics/backend/hogql_queries/web_vitals_paths_lazy_precompute.py— vitals lazy pathposthog/clickhouse/preaggregation/web_vitals_paths_preaggregated_sql.py— vitals schemaproducts/web_analytics/backend/hogql_queries/web_trends.py+web_trends_lazy_precompute.py— trend tiles lazy pathproducts/web_analytics/backend/hogql_queries/web_calendar_heatmap.py— Active Hours lazy pathproducts/analytics_platform/backend/lazy_computation/— framework + CONSISTENCY.md + READMEproducts/web_analytics/dags/eager_web_analytics_precompute.py— hourly baseline pre-warmer