Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(ci): record why the uv cache stays pruned (#124016)

Georges-Antoine Assib4641f69d806
diff --git a/docs/internal/ci-things-already-tried.md b/docs/internal/ci-things-already-tried.md
index 5622a1e51374..ae607e2aa1e4 100644
--- a/docs/internal/ci-things-already-tried.md
+++ b/docs/internal/ci-things-already-tried.md
@@ -496,6 +496,31 @@ A checkout of `pull_request.head.sha` also took 21 to 22 s, so the refspec is no
_Also asked as:_ slow checkout in the Django shards, head.ref or head.sha, why is the Django checkout slower than the product checkout
+### Keep the pre-built wheels in the uv cache with `prune-cache: false`
+
+**Verdict: rejected** · Oct 2026 · [Depot run](https://depot.dev/orgs/ntsdt08fpt/workflows/4qpth2kdws), [#121208](https://github.com/PostHog/posthog/pull/121208)
+
+`setup-uv` prunes the cache, so each backend shard downloads its wheels from PyPI in `uv sync`.
+Between 00:14 and 03:07 UTC on 27 Oct 2026, PyPI downloads from the Depot CI sandboxes were very slow.
+The `Install Python dependencies` step has `timeout-minutes: 5` and `retry: 1`.
+Both attempts stalled, and Depot killed the step with `exec-step child exited with signal SIGTERM and produced no result`.
+Each shard failed about 10.5 minutes after it started. About 25 Backend CI runs failed. From 22:00 UTC the evening before to 12:00 UTC that day, no other run failed this way.
+
+The proposal was to stop the pruning, so that the shards get the wheels from Depot Cache and not from PyPI.
+The cost was estimated, not measured, because no job runs with an unpruned cache:
+
+- The unpruned cache is about 1 GB. Depot Cache storage costs $0.20 per GB-month, so the storage cost is small.
+- Each shard restores the 1 GB in about 20 s (from #121208). `uv sync` from PyPI takes 11 to 23 s in a healthy shard. The estimated net cost is 5 to 20 s for each shard, on every run.
+- At $0.006 per minute for a 2 vCPU sandbox, the compute of the whole outage, with the killed shards and the full reruns, is equal to about 2 to 8 days of that extra restore time.
+
+The pruning stays because one outage at night costs less than the restore time.
+Try this again if PyPI outages occur during working hours or more than a few times each month. Then the developer time and the merge queue kicks change the result.
+A longer step timeout does not help: the 5-minute bound and the retry let the shard fail fast, and this outage was 3 hours long.
+
+`setup-uv` v9 changed the `prune-cache` default to `false`. Each cache-enabled call sets `prune-cache: true` so that the cache stays at about 3 MB.
+
+_Also asked as:_ uv sync timeout, uv sync SIGTERM, PyPI is slow in CI, cache the Python wheels, why is prune-cache true, exec-step child exited with signal SIGTERM
+
### Jest reports the Rust snapshots as obsolete
**Verdict: superseded** · Jan 2026 · [#46008](https://github.com/PostHog/posthog/pull/46008)