:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(aio): give clustering compute a third attempt to wait out clickhouse slot bursts (#120936)
Bernat Torres20994aeb0ca6
diff --git a/posthog/temporal/ai_observability/trace_clustering/README.md b/posthog/temporal/ai_observability/trace_clustering/README.mdindex dce627cd8bd4..bada029ee6ea 100644--- a/posthog/temporal/ai_observability/trace_clustering/README.md+++ b/posthog/temporal/ai_observability/trace_clustering/README.md@@ -398,7 +398,7 @@ Key constants in `constants.py`:## Error Handling- **Insufficient data**: Skip clustering if fewer than 1000 items (per analysis level)-- **Compute activity retries**: 2 attempts with exponential backoff. `ClickHouseAtCapacity` and `ConcurrencyLimitExceeded` wait a jittered 20–60s before the retry instead, and reach error tracking only on the last attempt+- **Compute activity retries**: 3 attempts with exponential backoff, capped by the 720s schedule-to-close budget. `ClickHouseAtCapacity` and `ConcurrencyLimitExceeded` wait a jittered 20–60s before the retry instead, and reach error tracking only on the last attempt- **LLM activity retries**: 2 attempts with longer intervals (for rate limits)- **Emit activity retries**: 3 attempts with exponential backoff- **LLM fallback**: If label generation fails, use "Cluster N" as titlediff --git a/posthog/temporal/ai_observability/trace_clustering/constants.py b/posthog/temporal/ai_observability/trace_clustering/constants.pyindex f82fe1ba3a12..1f829f6641d8 100644--- a/posthog/temporal/ai_observability/trace_clustering/constants.py+++ b/posthog/temporal/ai_observability/trace_clustering/constants.py@@ -53,16 +53,19 @@ EMIT_HEARTBEAT_TIMEOUT = timedelta(seconds=30) # 30 seconds - ClickHouse writes# Schedule-to-close timeouts - caps total time including all retry attempts,# backoff intervals, and queue time. Prevents runaway retries from blocking# the workflow indefinitely.-COMPUTE_SCHEDULE_TO_CLOSE_TIMEOUT = timedelta(seconds=720) # Two attempts, 60s backoff, and queue time+# A query-capacity rejection fails in about 10s (the limiter's wait) and backs off at most 60s, so+# two of them still leave a full 300s attempt inside this budget.+COMPUTE_SCHEDULE_TO_CLOSE_TIMEOUT = timedelta(seconds=720)LLM_SCHEDULE_TO_CLOSE_TIMEOUT = timedelta(seconds=1260) # 21 min: 2 attempts * 600s + backoff, so a timed-out first attempt still leaves the retry a full runAGGREGATES_SCHEDULE_TO_CLOSE_TIMEOUT = timedelta(seconds=330) # 5.5 min (1 attempt only, best-effort)EMIT_SCHEDULE_TO_CLOSE_TIMEOUT = timedelta(seconds=150) # 2.5 min (2 attempts * 60s + backoff)-# Compute activity - CPU bound, quick retries+# Compute activity - CPU bound, quick retries. A third attempt lets a run outlast a short burst in which+# no ClickHouse slot frees up; COMPUTE_SCHEDULE_TO_CLOSE_TIMEOUT still caps the total.COMPUTE_ACTIVITY_RETRY_POLICY = RetryPolicy(- maximum_attempts=2,+ maximum_attempts=3,initial_interval=timedelta(seconds=1),maximum_interval=timedelta(seconds=10),backoff_coefficient=2.0,