Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(ci): bound stalled cache restores in backend ci (#123431)

Raúl Negrón-Otero373397b03ba0
diff --git a/.depot/actions/pnpm-install/action.yml b/.depot/actions/pnpm-install/action.yml
index c2e92050eb81..12a17f4de700 100644
--- a/.depot/actions/pnpm-install/action.yml
+++ b/.depot/actions/pnpm-install/action.yml
@@ -1,6 +1,7 @@
# Depot mirror of .github/actions/pnpm-install (same pattern as .depot/actions/setup-sqlx-cli).
-# Kept equivalent to canonical except for the `inputs.install` guards and the master-gated
-# "Save pnpm store" step below. actions/cache routes to Depot Cache, so shared keys do not
+# Kept equivalent to canonical except for the `inputs.install` guards, the master-gated
+# "Save pnpm store" step below, and the timeout-minutes on both restores, which GitHub Actions
+# rejects in a composite step. actions/cache routes to Depot Cache, so shared keys do not
# collide.
name: Install pnpm dependencies
description: |
@@ -55,6 +56,8 @@ runs:
- name: Restore pnpm store
id: restore
if: inputs.install != 'false'
+ timeout-minutes: 2
+ continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: ${{ steps.pnpm-store.outputs.path }}
@@ -75,6 +78,8 @@ runs:
- name: Restore pnpm lockfile verdict
id: restore-verdict
if: inputs.install != 'false'
+ timeout-minutes: 2
+ continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: ${{ steps.pnpm-cache.outputs.path }}/lockfile-verified.jsonl
diff --git a/.depot/actions/setup-pnpm/action.yml b/.depot/actions/setup-pnpm/action.yml
index a0b40db31572..b0aaee6c223d 100644
--- a/.depot/actions/setup-pnpm/action.yml
+++ b/.depot/actions/setup-pnpm/action.yml
@@ -69,6 +69,9 @@ runs:
- name: Restore Node tool cache
id: node-toolcache
if: steps.node-version.outputs.version != ''
+ # GitHub Actions rejects timeout-minutes in a composite step, so only this mirror bounds the restore.
+ timeout-minutes: 2
+ continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: ${{ runner.tool_cache }}/node/${{ steps.node-version.outputs.version }}
diff --git a/.depot/actions/setup-python-cached/action.yml b/.depot/actions/setup-python-cached/action.yml
index 68852daa27ba..aa8cf86b3ced 100644
--- a/.depot/actions/setup-python-cached/action.yml
+++ b/.depot/actions/setup-python-cached/action.yml
@@ -1,3 +1,5 @@
+# Depot mirror of .github/actions/setup-python-cached. Only the tool cache restore differs. It sets
+# timeout-minutes, which GitHub Actions rejects in a composite step.
name: Set up Python
description: |
Wraps actions/setup-python with a pre-seeded tool cache
@@ -41,6 +43,8 @@ runs:
- name: Restore Python tool cache
id: python-toolcache
if: steps.python-version.outputs.version != ''
+ timeout-minutes: 2
+ continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: ${{ runner.tool_cache }}/Python/${{ steps.python-version.outputs.version }}
diff --git a/.depot/actions/setup-sqlx-cli/action.yml b/.depot/actions/setup-sqlx-cli/action.yml
index 21d8425ed8ef..3b8e226d555b 100644
--- a/.depot/actions/setup-sqlx-cli/action.yml
+++ b/.depot/actions/setup-sqlx-cli/action.yml
@@ -1,3 +1,5 @@
+# Depot mirror of .github/actions/setup-sqlx-cli. Only the cache restore differs. It sets
+# timeout-minutes, which GitHub Actions rejects in a composite step.
name: Setup sqlx-cli
description: |
Installs sqlx-cli and adds it to PATH.
@@ -20,6 +22,8 @@ runs:
steps:
- name: Restore sqlx-cli binary from cache
id: cache-sqlx-cli
+ timeout-minutes: 2
+ continue-on-error: true
uses: actions/cache/restore@9255dc7a253b0ccc959486e2bca901246202afeb # v5.0.1
with:
# The cache version hashes this path string, and runner.temp differs between
diff --git a/.depot/workflows/ci-backend.yml b/.depot/workflows/ci-backend.yml
index 1de9f907e9ab..b17e3ad6a1dc 100644
--- a/.depot/workflows/ci-backend.yml
+++ b/.depot/workflows/ci-backend.yml
@@ -38,6 +38,13 @@
# Without those plans, shards use the combined .test_durations file.
# - actions/cache uses Depot Cache, which Depot CI and Depot GitHub Actions runners share.
# GitHub-hosted runners use a separate GitHub Actions cache.
+# - A stalled actions/cache request holds a restore for about 15 minutes and then misses. The
+# client tries each request 5 times with a 3-minute socket timeout, and no input or env var
+# shortens either. SEGMENT_DOWNLOAD_TIMEOUT_MINS covers only the blob download. Restores here
+# and in the Depot composite mirrors set timeout-minutes and continue-on-error, so a stall
+# becomes an early miss. Schema restores get 4 minutes, so one retry runs before a
+# from-scratch migrate. Saves stay unbounded, because a timeout drops what they write. The uv
+# cache inside astral-sh/setup-uv stays unbounded, because a timeout there kills the install.
# - The env block clears GITHUB_TOKEN and OTEL_SERVICE_NAME because Depot provides them by default.
# - Token steps read vars.* and secrets.* from Depot's store. If a value is missing there,
# setup-action steps with continue-on-error fall back to the default GITHUB_TOKEN, a Depot
@@ -780,6 +787,8 @@ jobs:
enable-cache: false
continue-on-error: true
- name: Restore test durations from cache
+ timeout-minutes: 2
+ continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: .test_durations
@@ -807,6 +816,8 @@ jobs:
gh run download "$run_id" --repo "$GITHUB_REPOSITORY" --name test-durations --dir . \
|| echo "::warning::Timing artifact of run $run_id unavailable, sharding without durations"
- name: Restore per-segment test durations from cache
+ timeout-minutes: 2
+ continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: |
@@ -819,6 +830,8 @@ jobs:
# The timing workflow rebuilds it from the newest hourly recording. A miss
# leaves the file absent, and the events_json rows then run every listed test.
if: github.event_name == 'pull_request'
+ timeout-minutes: 2
+ continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: events-schema-prune.json
@@ -942,6 +955,8 @@ jobs:
steps:
- name: Look up schema cache
id: schema-cache
+ timeout-minutes: 4
+ continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: schema.sql.gz
@@ -1151,6 +1166,8 @@ jobs:
|| { echo "::group::compose launch log"; cat "${RUNNER_TEMP:-/tmp}/compose-launch.log" 2>/dev/null || true; echo "::endgroup::"; exit 1; }
- name: Restore schema cache from master
if: ${{ github.event_name == 'pull_request' }}
+ timeout-minutes: 4
+ continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: schema.sql.gz
@@ -1207,7 +1224,7 @@ jobs:
bin/wait-for-docker temporal
PYTHONFAULTHANDLER=1 timeout --signal=ABRT --kill-after=30s 3m python manage.py register_temporal_search_attributes
- name: Restore test durations from cache
- timeout-minutes: 7
+ timeout-minutes: 2
continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
@@ -1800,6 +1817,8 @@ jobs:
- name: Restore master schema dump from cache
id: master-schema-cache
if: github.event_name != 'push' && steps.master-schema-key.outputs.key != ''
+ timeout-minutes: 4
+ continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: schema.sql.gz
@@ -3144,6 +3163,8 @@ jobs:
# Restore before setup-python so the tool cache is warm when it starts.
- name: Restore Python tool cache
id: python-toolcache
+ timeout-minutes: 2
+ continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: ${{ runner.tool_cache }}/Python/${{ matrix.python-version }}
@@ -3293,6 +3314,8 @@ jobs:
|| { echo "::group::compose launch log"; cat "${RUNNER_TEMP:-/tmp}/compose-launch.log" 2>/dev/null || true; echo "::endgroup::"; exit 1; }
- name: Restore schema cache from master
if: ${{ github.event_name == 'pull_request' }}
+ timeout-minutes: 4
+ continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: schema.sql.gz
@@ -3344,6 +3367,8 @@ jobs:
admin_psql -c "DROP DATABASE IF EXISTS test_posthog;" || true
fi
- name: Restore test durations from cache
+ timeout-minutes: 2
+ continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: .test_durations
@@ -3371,6 +3396,8 @@ jobs:
gh run download "$run_id" --repo "$GITHUB_REPOSITORY" --name test-durations --dir . \
|| echo "::warning::Timing artifact of run $run_id unavailable, sharding without durations"
- name: Restore per-segment test durations from cache
+ timeout-minutes: 2
+ continue-on-error: true
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: |
diff --git a/tools/hogli-commands/hogli_commands/depot_mirrors.py b/tools/hogli-commands/hogli_commands/depot_mirrors.py
index bc3200ebc6c2..f1917ceac52e 100644
--- a/tools/hogli-commands/hogli_commands/depot_mirrors.py
+++ b/tools/hogli-commands/hogli_commands/depot_mirrors.py
@@ -13,7 +13,7 @@ from pathlib import Path
# These mirrors carry intentional deltas that each mirror's header documents.
# Every other mirror stays byte-identical to its canonical action.
-DELTA_MIRRORS = frozenset({"commit-snapshots", "pnpm-install", "setup-pnpm"})
+DELTA_MIRRORS = frozenset({"commit-snapshots", "pnpm-install", "setup-pnpm", "setup-python-cached", "setup-sqlx-cli"})
# Depot resolves both `./.github/actions/<name>` and `./.depot/actions/<name>` from .depot/actions/<name>.
LOCAL_ACTION_USE = re.compile(r"""uses:\s*['"]?\./\.(?:github|depot)/actions/([A-Za-z0-9_.-]+)""")