Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(aio): give clustering compute a third attempt to wait out clickhouse slot bursts (#120936)

Bernat Torres20994aeb0ca6
diff --git a/posthog/temporal/ai_observability/trace_clustering/README.md b/posthog/temporal/ai_observability/trace_clustering/README.md
index dce627cd8bd4..bada029ee6ea 100644
--- a/posthog/temporal/ai_observability/trace_clustering/README.md
+++ b/posthog/temporal/ai_observability/trace_clustering/README.md
@@ -398,7 +398,7 @@ Key constants in `constants.py`:
## Error Handling
- **Insufficient data**: Skip clustering if fewer than 1000 items (per analysis level)
-- **Compute activity retries**: 2 attempts with exponential backoff. `ClickHouseAtCapacity` and `ConcurrencyLimitExceeded` wait a jittered 20–60s before the retry instead, and reach error tracking only on the last attempt
+- **Compute activity retries**: 3 attempts with exponential backoff, capped by the 720s schedule-to-close budget. `ClickHouseAtCapacity` and `ConcurrencyLimitExceeded` wait a jittered 20–60s before the retry instead, and reach error tracking only on the last attempt
- **LLM activity retries**: 2 attempts with longer intervals (for rate limits)
- **Emit activity retries**: 3 attempts with exponential backoff
- **LLM fallback**: If label generation fails, use "Cluster N" as title
diff --git a/posthog/temporal/ai_observability/trace_clustering/constants.py b/posthog/temporal/ai_observability/trace_clustering/constants.py
index f82fe1ba3a12..1f829f6641d8 100644
--- a/posthog/temporal/ai_observability/trace_clustering/constants.py
+++ b/posthog/temporal/ai_observability/trace_clustering/constants.py
@@ -53,16 +53,19 @@ EMIT_HEARTBEAT_TIMEOUT = timedelta(seconds=30) # 30 seconds - ClickHouse writes
# Schedule-to-close timeouts - caps total time including all retry attempts,
# backoff intervals, and queue time. Prevents runaway retries from blocking
# the workflow indefinitely.
-COMPUTE_SCHEDULE_TO_CLOSE_TIMEOUT = timedelta(seconds=720) # Two attempts, 60s backoff, and queue time
+# A query-capacity rejection fails in about 10s (the limiter's wait) and backs off at most 60s, so
+# two of them still leave a full 300s attempt inside this budget.
+COMPUTE_SCHEDULE_TO_CLOSE_TIMEOUT = timedelta(seconds=720)
LLM_SCHEDULE_TO_CLOSE_TIMEOUT = timedelta(
seconds=1260
) # 21 min: 2 attempts * 600s + backoff, so a timed-out first attempt still leaves the retry a full run
AGGREGATES_SCHEDULE_TO_CLOSE_TIMEOUT = timedelta(seconds=330) # 5.5 min (1 attempt only, best-effort)
EMIT_SCHEDULE_TO_CLOSE_TIMEOUT = timedelta(seconds=150) # 2.5 min (2 attempts * 60s + backoff)
-# Compute activity - CPU bound, quick retries
+# Compute activity - CPU bound, quick retries. A third attempt lets a run outlast a short burst in which
+# no ClickHouse slot frees up; COMPUTE_SCHEDULE_TO_CLOSE_TIMEOUT still caps the total.
COMPUTE_ACTIVITY_RETRY_POLICY = RetryPolicy(
- maximum_attempts=2,
+ maximum_attempts=3,
initial_interval=timedelta(seconds=1),
maximum_interval=timedelta(seconds=10),
backoff_coefficient=2.0,