Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

4.9 KB
Raw Blame

Batch workflow audience resolution

How a batch-triggered workflow run resolves its audience, and the knobs that bound it.

The resolver

Triggering a batch workflow creates one cyclotron job on HOGFLOW_BATCH_RESOLVE_QUEUE, processed by the cdp-cyclotron-worker-batch-resolve consumer. Each dequeue fetches one audience page from Django's internal endpoints (user_blast_radius_persons for person audiences, account_audience for account audiences), enqueues that page's runs, and reschedules itself with the next cursor until the audience is exhausted or truncated by maxAudienceSize. The terminal status (completed/failed) is PUT back to Django on a final dequeue, and the resolver only acks after that write succeeds.

Audience fetch timeout

The page fetch runs a ClickHouse query, so it does not fit the generic 3s EXTERNAL_REQUEST_TIMEOUT_MS inter-service budget:

  • CDP_HOG_FLOW_BATCH_AUDIENCE_FETCH_TIMEOUT_MS (default 30000) — client-side budget in milliseconds for each audience fetch (blast-radius count, persons page, account page).

A fetch that exceeds the budget aborts, retries up to MAX_RESOLVER_ATTEMPTS times with backoff, and then the run is marked failed on the workflow's log stream. A timeout gets its own terminal reason, Batch resolver failed: Audience query timed out…, which names the budget and the attempt count. Any other fetch failure keeps the generic Batch resolver failed: Audience fetch failed permanently…, so search for both texts when you triage a failed run. Keep the budget under the HogQL default max_execution_time (60s): above it, the client only waits longer for a query ClickHouse will kill anyway. Note the client abort does not cancel the ClickHouse query — a query slower than the budget keeps running server-side until the HogQL cap, so a too-small budget wastes a full query execution per attempt.

Lock heartbeats and batch size

A near-budget fetch holds one job for ~30s, the same magnitude as the janitor's stall threshold (CYCLOTRON_NODE_JANITOR_STALL_TIMEOUT_MS, default 30s). Two things keep the janitor from reclaiming a healthy job:

  • The consumer heartbeats the held job every 10s while it processes.
  • It dequeues one job at a time (batchMaxSize: 1). Pages are processed serially, so a bigger batch adds no throughput — it only leaves queued peers un-heartbeated behind a slow fetch.

Lease heartbeats update the job's database lock; they do not refresh the worker's poll health check. The worker waits for page processing to finish before polling again. Its heartbeatTimeoutMs adds 30 seconds to the audience fetch budget for processing and monitoring flushes. The default health timeout is 60 seconds. This keeps an allowed slow fetch from reporting the worker as unhealthy while its lease remains valid.

Observing

  • Fetch durations: instrumented_function_duration_seconds for cdpBatchResolve.getBlastRadiusPersons and cdpBatchResolve.getAccountAudiencePage. Read the bucket counts rather than a quantile before tuning either. The finite bucket edges are 0.025, 0.1, 0.4, 1.6, 6.4, 25.6 and 102.4 seconds, so a quantile between two edges reports the share of slow fetches and not how long they took. CdpBatchAudienceFetchSlow counts fetches slower than 25.6s, the edge closest to the budget, and fires before any fetch aborts. That alert reads the persons function only. A slow account page therefore gets no warning before it starts to time out, so watch its bucket counts by hand or widen the alert.
  • Timeouts: cdp_batch_hog_flow_audience_fetch_timeout{endpoint} counts each fetch that used its full budget and aborted. A timeout raises AudienceFetchTimeoutError, which is a separate type from a transport failure, so the retry logs and the customer-visible failure reason name the timeout. CdpBatchAudienceFetchTimingOut alerts on a sustained rate. Both alerts live in PostHog/charts alerts/specs/cdp.yaml.
  • Failures: cdp_batch_hog_flow_resolver_pages_processed{outcome="fetch_failure"}, and a Batch resolver failed: <reason> row in ClickHouse log_entries with log_source = 'hog_flow' and log_source_id = the batch job id (not the workflow id).

Known gap: the failure reason is not visible in the app

The resolver keys its failure, audience-truncation, and cancel logs by batch job id, but every in-app log reader queries by workflow id (LogsViewer sourceId={workflow.id}; the Logs panel of a batch workflow redirects to Invocations). Normal run logs use log_source_id = workflow id and instance_id = run id. So a batch run that fails before enrolling anyone shows only a failed badge, and the reason is only reachable by querying log_entries directly. Re-keying those three resolver log sites to log_source_id: state.hogFlowId, instance_id: state.batchJobId would make them appear in the workflow's Logs view and as the batch job's own log stream; that is a separate change from the timeout.