:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(ci): record why the uv cache stays pruned (#124016)
Georges-Antoine Assib4641f69d806
diff --git a/docs/internal/ci-things-already-tried.md b/docs/internal/ci-things-already-tried.mdindex 5622a1e51374..ae607e2aa1e4 100644--- a/docs/internal/ci-things-already-tried.md+++ b/docs/internal/ci-things-already-tried.md@@ -496,6 +496,31 @@ A checkout of `pull_request.head.sha` also took 21 to 22 s, so the refspec is no_Also asked as:_ slow checkout in the Django shards, head.ref or head.sha, why is the Django checkout slower than the product checkout+### Keep the pre-built wheels in the uv cache with `prune-cache: false`++**Verdict: rejected** · Oct 2026 · [Depot run](https://depot.dev/orgs/ntsdt08fpt/workflows/4qpth2kdws), [#121208](https://github.com/PostHog/posthog/pull/121208)++`setup-uv` prunes the cache, so each backend shard downloads its wheels from PyPI in `uv sync`.+Between 00:14 and 03:07 UTC on 27 Oct 2026, PyPI downloads from the Depot CI sandboxes were very slow.+The `Install Python dependencies` step has `timeout-minutes: 5` and `retry: 1`.+Both attempts stalled, and Depot killed the step with `exec-step child exited with signal SIGTERM and produced no result`.+Each shard failed about 10.5 minutes after it started. About 25 Backend CI runs failed. From 22:00 UTC the evening before to 12:00 UTC that day, no other run failed this way.++The proposal was to stop the pruning, so that the shards get the wheels from Depot Cache and not from PyPI.+The cost was estimated, not measured, because no job runs with an unpruned cache:++- The unpruned cache is about 1 GB. Depot Cache storage costs $0.20 per GB-month, so the storage cost is small.+- Each shard restores the 1 GB in about 20 s (from #121208). `uv sync` from PyPI takes 11 to 23 s in a healthy shard. The estimated net cost is 5 to 20 s for each shard, on every run.+- At $0.006 per minute for a 2 vCPU sandbox, the compute of the whole outage, with the killed shards and the full reruns, is equal to about 2 to 8 days of that extra restore time.++The pruning stays because one outage at night costs less than the restore time.+Try this again if PyPI outages occur during working hours or more than a few times each month. Then the developer time and the merge queue kicks change the result.+A longer step timeout does not help: the 5-minute bound and the retry let the shard fail fast, and this outage was 3 hours long.++`setup-uv` v9 changed the `prune-cache` default to `false`. Each cache-enabled call sets `prune-cache: true` so that the cache stays at about 3 MB.++_Also asked as:_ uv sync timeout, uv sync SIGTERM, PyPI is slow in CI, cache the Python wheels, why is prune-cache true, exec-step child exited with signal SIGTERM+### Jest reports the Rust snapshots as obsolete**Verdict: superseded** · Jan 2026 · [#46008](https://github.com/PostHog/posthog/pull/46008)