Skip to content

Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.

CI: keep one generation of prefix-restored caches, share Core's pip cache (#14263)

Daniel Hane632df1ca6e6
diff --git a/.github/workflows/cache-janitor.yml b/.github/workflows/cache-janitor.yml
index d76e84a62a8..4880fd00433 100644
--- a/.github/workflows/cache-janitor.yml
+++ b/.github/workflows/cache-janitor.yml
@@ -48,7 +48,7 @@ name: Cache janitor
# 2. Superseded generations of families whose keys embed a build identity:
# codeql-overlay-base-database-* keys embed commit SHA + run id; keep 1
# codeql-trap-* keys embed commit SHA; keep 1
-# v0-rust-* Swatinem/rust-cache
+# v0-rust-* Swatinem/rust-cache; keep 1
#
# CodeQL default setup writes a new overlay-base database (420 MB python
# + 330 MB javascript) on EVERY push to main, keyed by commit SHA, and a
@@ -88,6 +88,16 @@ name: Cache janitor
# can answer nothing -- unreachable refs, generations past keep, and entries on an
# open PR that nothing has read in open_pr_idle_days.
#
+# pip-v2-*, uv-* and v0-rust-* keep 1 regardless of `keep`. All three restore with
+# a prefix fallback, so a run whose exact key is gone still starts from the newest
+# generation and pays only for what changed; the second copy answers nothing but
+# an exact revert. It cost the most: on 2026-10-19 the repo sat at 49.1 of 50 GiB,
+# and second generations of those three families held 14.8 GiB of it (pip 8.8,
+# uv 3.1, rust 3.0), mostly because seven pip families key on the same
+# pyproject.toml + studio/backend/requirements/*.txt and every edit to those
+# files mints a new generation in all seven at once. `keep` now only governs
+# fe-dist, the one ranked family without a prefix fallback.
+#
# Everything else is left alone. The hf-* / *-gguf-* model caches use exact keys,
# so an entry that looks superseded is the only one its key will ever match and
# deleting it costs a multi-GB re-download.
@@ -105,7 +115,7 @@ on:
options: [report, delete]
default: report
keep:
- description: 'Generations kept per prefix. 2 so an in-flight run cannot lose the cache it just resolved, and so a reverted dependency bump still hits.'
+ description: 'Generations kept per fe-dist prefix (pip-v2, uv, rust and CodeQL always keep 1). 2 so a reverted frontend change still hits.'
type: string
default: '2'
open_pr_idle_days:
@@ -303,8 +313,9 @@ jobs:
while IFS=$'\t' read -r id created size ref ver key accessed; do
[ -z "${id:-}" ] && continue
# Per-family keep. `keep` (the input) is for families where an older
- # generation can still answer an exact-key restore; CodeQL restores
- # by prefix and takes the newest, so its second copy answers nothing.
+ # generation can still answer a restore nothing newer can. CodeQL,
+ # pip-v2, uv and rust-cache all restore by prefix and take the newest,
+ # so their second copy answers only an exact revert (see the header).
fkeep="$KEEP"
case "$key" in
codeql-overlay-base-database-*)
@@ -312,36 +323,24 @@ jobs:
codeql-trap-*)
pre=$(printf '%s' "$key" | sed -E 's/-[0-9a-f]{40}$//'); fkeep=1 ;;
v0-rust-*)
- pre=$(printf '%s' "$key" | sed -E 's/-[0-9a-f]{8,}$//') ;;
- pip-v2-notebooks-colab-*)
- # One generation, not `keep`. This family replays a 697-pin Colab
- # freeze and measures 6.97 GB per generation, so two copies of one
- # job's downloads held 13.95 GB, 30% of the repo's whole 50 GiB
- # budget (2026-09-18, with the repo at 92% full and evicting).
- # Every other pip family averages 0.5 GB.
- #
- # The general argument for keep=2 is that the previous generation
- # still answers an exact-key restore. It does here too, but only
- # for an exact revert of colab_pip_freeze.gpu.txt or of
- # notebooks-ci.yml, which is rarer than the eviction pressure the
- # second copy causes on every other family. Same trade as the
- # CodeQL rule above: keep is a budget dial, and this family spends
- # more of it than the hit it buys is worth.
- #
- # The glob is deliberately a prefix and not the exact job name:
- # it has to keep matching `notebooks-colab-cpu`, and any further
- # suffix, because retiring the prefix is how that job collects a
- # trim at all (see the comment on its `name` in notebooks-ci.yml).
- # Each retired prefix is then its own family here, so keep=1
- # drains it to one entry and the 7-day idle timer takes the last.
- pre=$(printf '%s' "$key" | sed -E 's/-[0-9a-f]{64}$//'); fkeep=1 ;;
+ # rust-cache's restore-key is the key minus the trailing lockfile
+ # hash, i.e. exactly this prefix, so a lockfile bump still starts
+ # from the surviving generation.
+ pre=$(printf '%s' "$key" | sed -E 's/-[0-9a-f]{8,}$//'); fkeep=1 ;;
pip-v2-*|uv-*|fe-dist-*)
# Strip only the trailing dependency hash. Everything before it --
# the job name for pip-v2, the OS for uv and fe-dist -- stays in the
# prefix on purpose: those are separate keys that never substitute
# for one another, and folding them into one group would rank live
- # caches as generations of each other and delete all but two.
- pre=$(printf '%s' "$key" | sed -E 's/-[0-9a-f]{64}$//') ;;
+ # caches as generations of each other and delete all but one.
+ #
+ # A retired prefix (notebooks-colab -> notebooks-colab-cpu, see its
+ # `name` in notebooks-ci.yml) is its own family here, so keep=1
+ # drains it to one entry and the 7-day idle timer takes the last.
+ pre=$(printf '%s' "$key" | sed -E 's/-[0-9a-f]{64}$//')
+ # fe-dist has no restore-keys: an older generation is the only
+ # thing an exact revert of the frontend can hit, so it keeps `keep`.
+ case "$key" in fe-dist-*) ;; *) fkeep=1 ;; esac ;;
*) continue ;;
esac
# Unshortened means the suffix did not match, which would make
@@ -381,7 +380,7 @@ jobs:
echo "| metric | value |"
echo "| --- | --- |"
echo "| mode | $([ "$DELETE" = yes ] && echo delete || echo 'report only') |"
- echo "| kept per prefix | $KEEP |"
+ echo "| kept per prefix | 1 (fe-dist: $KEEP) |"
echo "| caches total | $count |"
echo "| dead-family candidates | $dead |"
echo "| closed-PR candidates | $stale_pr |"
@@ -396,5 +395,5 @@ jobs:
# 80% of the 50 GiB ceiling. Above this a single dependency bump can
# push the repo over between two scheduled sweeps.
if [ "$after" -gt 42949672960 ]; then
- echo "::warning::Cache usage is $(gib "$after") GiB of 50 GiB after pruning. Lower keep to 1, or find the family that grew."
+ echo "::warning::Cache usage is $(gib "$after") GiB of 50 GiB after pruning. Find the family that grew."
fi
diff --git a/.github/workflows/consolidated-tests-ci.yml b/.github/workflows/consolidated-tests-ci.yml
index 1d5ac4dc0d6..1dcb6be7ec7 100644
--- a/.github/workflows/consolidated-tests-ci.yml
+++ b/.github/workflows/consolidated-tests-ci.yml
@@ -2364,19 +2364,16 @@ jobs:
- name: Restore the pip cache
id: pip-cache
- # Its own `name`, not `consolidated`'s. Two jobs sharing one is the
- # defect tests/studio/test_pip_cache_naming.py exists to reject: the
- # save is gated on `cache-hit != 'true'`, so whichever job finished
- # first on main would write the entry and the other would restore it
- # and never save its own. The key-files are the same because the
- # install is the same, which that file also allows for explicitly.
- #
- # The first run on a pull request therefore misses and downloads. The
- # entry is written by the next push to main and warm from then on, the
- # same lifecycle every other key here has.
+ # `consolidated`'s entry, read-only: this job has no pip-cache-save. The
+ # install is the same core-cpu-setup call on the same matrix (asserted by
+ # tests/studio/test_zoo_suite_parallel_isolation.py), so a separate entry
+ # held the same ~1.4 GiB of wheels twice. A shared name is only a defect
+ # when both jobs save, since then the first to finish wins and the other's
+ # extras are never written; with one writer there is nothing to lose.
+ # tests/studio/test_pip_cache_naming.py allows exactly this borrower.
uses: ./.github/actions/pip-cache-restore
with:
- name: consolidated-zoo
+ name: consolidated
key-files: |
pyproject.toml
studio/backend/requirements/*.txt
@@ -2513,14 +2510,6 @@ jobs:
echo " - unsloth_zoo @ ${UNSLOTH_ZOO_REF} pytest tests/ (GPU-only cases deselected)"
echo " - unsloth_zoo.compiler.test_apply_fused_lm_head"
- - name: Save the pip cache
- if: always()
- uses: ./.github/actions/pip-cache-save
- with:
- dir: ${{ steps.pip-cache.outputs.dir }}
- key: ${{ steps.pip-cache.outputs.key }}
- cache-hit: ${{ steps.pip-cache.outputs.cache-hit }}
-
llama-cpp-smoke:
# Standalone llama.cpp build + smoke. Earlier this lived inside every
# consolidated matrix cell and re-cmake'd llama.cpp ~5 min per cell --
diff --git a/tests/studio/test_cache_budget_discipline.py b/tests/studio/test_cache_budget_discipline.py
index e52b7b4dca3..4a833637261 100644
--- a/tests/studio/test_cache_budget_discipline.py
+++ b/tests/studio/test_cache_budget_discipline.py
@@ -17,8 +17,8 @@ WORKFLOWS = REPO / ".github" / "workflows"
# ungated on PR refs and evicts main's shared entry.
PIP_CACHE_JOBS = {
("consolidated-tests-ci.yml", "consolidated"),
- # Same install as `consolidated` but needs its own key: saves are gated on
- # `cache-hit != 'true'`, so a shared key lets only one job ever save.
+ # Same install as `consolidated`, so it restores that entry read-only and never
+ # saves; see READ_ONLY_BORROWERS in test_pip_cache_naming.py.
("consolidated-tests-ci.yml", "consolidated-zoo"),
("consolidated-tests-ci.yml", "llama-cpp-smoke"),
("mlx-ci.yml", "dispatch"),
@@ -35,6 +35,10 @@ PIP_CACHE_JOBS = {
# Partial saves on purpose: ccache checksums entries, so a truncated cache only costs misses.
PARTIAL_SAVE_JOBS = {("prebuilt-cuda-wheels.yml", "warm")}
+# Restore another job's pip cache and deliberately save nothing (mirrors
+# READ_ONLY_BORROWERS in test_pip_cache_naming.py, which checks the owner side).
+PIP_CACHE_READ_ONLY_JOBS = {("consolidated-tests-ci.yml", "consolidated-zoo")}
+
HEAVY = re.compile(
r"torch|transformers|trl|peft|vllm|bitsandbytes|sentence-transformers|diffusers"
r"|accelerate|datasets|requirements/"
@@ -408,6 +412,11 @@ def test_every_pip_cache_scopes_its_key_to_what_it_installs(name, jid):
def test_every_restored_pip_cache_is_also_saved_and_wired_to_its_restore(name, jid):
"""A restore without a save fills nothing; a save reading the wrong restore outputs writes nothing."""
restore, save = _pip_cache_steps(name, jid)
+ if (name, jid) in PIP_CACHE_READ_ONLY_JOBS:
+ assert (
+ restore is not None and save is None
+ ), f"{name}:{jid} is a read-only borrower: it must restore and must not save"
+ return
assert restore is not None and save is not None, (
f"{name}:{jid} has restore={restore is not None}, save={save is not None}; the "
f"pair has to stay together or the cache is never populated"
diff --git a/tests/studio/test_pip_cache_naming.py b/tests/studio/test_pip_cache_naming.py
index 4584a95c7c8..6502e53cbcc 100644
--- a/tests/studio/test_pip_cache_naming.py
+++ b/tests/studio/test_pip_cache_naming.py
@@ -15,6 +15,11 @@ WORKFLOWS = sorted((REPO / ".github" / "workflows").glob("*.yml"))
RESTORE_SUFFIX = "actions/pip-cache-restore"
SAVE_SUFFIX = "actions/pip-cache-save"
+# Jobs that restore another job's entry and never save. Only safe when the install
+# is identical, so a borrower's extras cannot exist; consolidated-zoo runs the same
+# core-cpu-setup on the same matrix, asserted by test_zoo_suite_parallel_isolation.py.
+READ_ONLY_BORROWERS = {"consolidated-tests-ci.yml:consolidated-zoo": "consolidated"}
+
def _jobs(path):
doc = yaml.safe_load(path.read_text(encoding = "utf-8")) or {}
@@ -58,6 +63,14 @@ def test_every_restore_names_itself(path):
assert files, f"{path.name}:{job} passes no key-files"
+def _saves(steps):
+ return [s for s in steps if str(s.get("uses", "")).endswith(SAVE_SUFFIX)]
+
+
+def _key_files(steps):
+ return " ".join(((_restores(steps)[0].get("with") or {}).get("key-files") or "").split())
+
+
def test_no_two_jobs_share_a_cache_name():
seen = {}
for path in WORKFLOWS:
@@ -65,6 +78,8 @@ def test_no_two_jobs_share_a_cache_name():
for step in _restores(steps):
name = (step.get("with") or {}).get("name") or ""
where = f"{path.name}:{job}"
+ if READ_ONLY_BORROWERS.get(where) == name:
+ continue
assert name not in seen, (
f"cache name {name!r} is used by both {seen[name]} and {where}. "
f"A shared name is a shared key: only the first job to finish on "
@@ -77,10 +92,13 @@ def test_jobs_sharing_key_files_still_have_distinct_names():
by_files = {}
for path in WORKFLOWS:
for job, steps in _jobs(path):
+ where = f"{path.name}:{job}"
for step in _restores(steps):
with_ = step.get("with") or {}
+ if READ_ONLY_BORROWERS.get(where) == with_.get("name"):
+ continue
files = " ".join((with_.get("key-files") or "").split())
- by_files.setdefault(files, []).append((f"{path.name}:{job}", with_.get("name")))
+ by_files.setdefault(files, []).append((where, with_.get("name")))
for files, entries in by_files.items():
names = [n for _, n in entries]
assert len(names) == len(set(names)), (
@@ -89,6 +107,32 @@ def test_jobs_sharing_key_files_still_have_distinct_names():
)
[email protected]("where,owner", sorted(READ_ONLY_BORROWERS.items()))
+def test_read_only_borrowers_never_save_and_their_owner_does(where, owner):
+ jobs = {f"{p.name}:{j}": steps for p in WORKFLOWS for j, steps in _jobs(p)}
+ assert where in jobs, f"{where} no longer exists; drop it from READ_ONLY_BORROWERS"
+ restores = _restores(jobs[where])
+ assert [(s.get("with") or {}).get("name") for s in restores] == [
+ owner
+ ], f"{where} must restore exactly one pip cache, {owner!r}'s"
+ assert not _saves(jobs[where]), (
+ f"{where} borrows {owner!r}'s entry, so it must not save: two writers of one key "
+ f"race, and the loser's wheels are never written"
+ )
+ owners = [
+ w
+ for w, steps in jobs.items()
+ if w != where
+ and _saves(steps)
+ and any((s.get("with") or {}).get("name") == owner for s in _restores(steps))
+ ]
+ assert len(owners) == 1, f"{owner!r} needs exactly one saving job, found {owners}"
+ assert _key_files(jobs[where]) == _key_files(jobs[owners[0]]), (
+ f"{where} keys on different files than {owners[0]}, so it would look up a key "
+ f"its owner never writes"
+ )
+
+
def test_key_is_versioned_and_carries_the_name():
action = (REPO / ".github/actions/pip-cache-restore/action.yml").read_text(encoding = "utf-8")
assert 'prefix="pip-v2-${name}-' in action, (
diff --git a/tests/studio/test_zoo_suite_parallel_isolation.py b/tests/studio/test_zoo_suite_parallel_isolation.py
index 621cd234779..c482b1b90df 100644
--- a/tests/studio/test_zoo_suite_parallel_isolation.py
+++ b/tests/studio/test_zoo_suite_parallel_isolation.py
@@ -286,11 +286,8 @@ def test_both_halves_of_core_share_one_install_preamble() -> None:
f"same environment, so their preambles have to be the same steps in the same order"
)
for left, right in zip(a, b):
- # A shared pip cache name is a shared key; see tests/studio/test_pip_cache_naming.py.
- left, right = dict(left), dict(right)
- if "pip-cache-restore" in str(left.get("uses", "")):
- left["with"] = {k: v for k, v in left["with"].items() if k != "name"}
- right["with"] = {k: v for k, v in right["with"].items() if k != "name"}
+ # The pip cache restore is compared whole, name included: consolidated-zoo
+ # borrows consolidated's entry read-only (see test_pip_cache_naming.py).
assert left == right, (
f"the Core preambles have drifted at step "
f"{left.get('name') or left.get('uses')!r}:\n consolidated: {left}\n"