Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.
Python agentaichatgptdeepseekfine-tuninggemmaimage-generationllamallmllmsopenaipythonqwenreinforcement-learningself-hostedstable-diffusiontext-to-speechttsuiunsloth
CI: keep one generation of prefix-restored caches, share Core's pip cache (#14263)
Daniel Hane632df1ca6e6
diff --git a/.github/workflows/cache-janitor.yml b/.github/workflows/cache-janitor.ymlindex d76e84a62a8..4880fd00433 100644--- a/.github/workflows/cache-janitor.yml+++ b/.github/workflows/cache-janitor.yml@@ -48,7 +48,7 @@ name: Cache janitor# 2. Superseded generations of families whose keys embed a build identity:# codeql-overlay-base-database-* keys embed commit SHA + run id; keep 1# codeql-trap-* keys embed commit SHA; keep 1-# v0-rust-* Swatinem/rust-cache+# v0-rust-* Swatinem/rust-cache; keep 1## CodeQL default setup writes a new overlay-base database (420 MB python# + 330 MB javascript) on EVERY push to main, keyed by commit SHA, and a@@ -88,6 +88,16 @@ name: Cache janitor# can answer nothing -- unreachable refs, generations past keep, and entries on an# open PR that nothing has read in open_pr_idle_days.#+# pip-v2-*, uv-* and v0-rust-* keep 1 regardless of `keep`. All three restore with+# a prefix fallback, so a run whose exact key is gone still starts from the newest+# generation and pays only for what changed; the second copy answers nothing but+# an exact revert. It cost the most: on 2026-10-19 the repo sat at 49.1 of 50 GiB,+# and second generations of those three families held 14.8 GiB of it (pip 8.8,+# uv 3.1, rust 3.0), mostly because seven pip families key on the same+# pyproject.toml + studio/backend/requirements/*.txt and every edit to those+# files mints a new generation in all seven at once. `keep` now only governs+# fe-dist, the one ranked family without a prefix fallback.+## Everything else is left alone. The hf-* / *-gguf-* model caches use exact keys,# so an entry that looks superseded is the only one its key will ever match and# deleting it costs a multi-GB re-download.@@ -105,7 +115,7 @@ on:options: [report, delete]default: reportkeep:- description: 'Generations kept per prefix. 2 so an in-flight run cannot lose the cache it just resolved, and so a reverted dependency bump still hits.'+ description: 'Generations kept per fe-dist prefix (pip-v2, uv, rust and CodeQL always keep 1). 2 so a reverted frontend change still hits.'type: stringdefault: '2'open_pr_idle_days:@@ -303,8 +313,9 @@ jobs:while IFS=$'\t' read -r id created size ref ver key accessed; do[ -z "${id:-}" ] && continue# Per-family keep. `keep` (the input) is for families where an older- # generation can still answer an exact-key restore; CodeQL restores- # by prefix and takes the newest, so its second copy answers nothing.+ # generation can still answer a restore nothing newer can. CodeQL,+ # pip-v2, uv and rust-cache all restore by prefix and take the newest,+ # so their second copy answers only an exact revert (see the header).fkeep="$KEEP"case "$key" incodeql-overlay-base-database-*)@@ -312,36 +323,24 @@ jobs:codeql-trap-*)pre=$(printf '%s' "$key" | sed -E 's/-[0-9a-f]{40}$//'); fkeep=1 ;;v0-rust-*)- pre=$(printf '%s' "$key" | sed -E 's/-[0-9a-f]{8,}$//') ;;- pip-v2-notebooks-colab-*)- # One generation, not `keep`. This family replays a 697-pin Colab- # freeze and measures 6.97 GB per generation, so two copies of one- # job's downloads held 13.95 GB, 30% of the repo's whole 50 GiB- # budget (2026-09-18, with the repo at 92% full and evicting).- # Every other pip family averages 0.5 GB.- #- # The general argument for keep=2 is that the previous generation- # still answers an exact-key restore. It does here too, but only- # for an exact revert of colab_pip_freeze.gpu.txt or of- # notebooks-ci.yml, which is rarer than the eviction pressure the- # second copy causes on every other family. Same trade as the- # CodeQL rule above: keep is a budget dial, and this family spends- # more of it than the hit it buys is worth.- #- # The glob is deliberately a prefix and not the exact job name:- # it has to keep matching `notebooks-colab-cpu`, and any further- # suffix, because retiring the prefix is how that job collects a- # trim at all (see the comment on its `name` in notebooks-ci.yml).- # Each retired prefix is then its own family here, so keep=1- # drains it to one entry and the 7-day idle timer takes the last.- pre=$(printf '%s' "$key" | sed -E 's/-[0-9a-f]{64}$//'); fkeep=1 ;;+ # rust-cache's restore-key is the key minus the trailing lockfile+ # hash, i.e. exactly this prefix, so a lockfile bump still starts+ # from the surviving generation.+ pre=$(printf '%s' "$key" | sed -E 's/-[0-9a-f]{8,}$//'); fkeep=1 ;;pip-v2-*|uv-*|fe-dist-*)# Strip only the trailing dependency hash. Everything before it --# the job name for pip-v2, the OS for uv and fe-dist -- stays in the# prefix on purpose: those are separate keys that never substitute# for one another, and folding them into one group would rank live- # caches as generations of each other and delete all but two.- pre=$(printf '%s' "$key" | sed -E 's/-[0-9a-f]{64}$//') ;;+ # caches as generations of each other and delete all but one.+ #+ # A retired prefix (notebooks-colab -> notebooks-colab-cpu, see its+ # `name` in notebooks-ci.yml) is its own family here, so keep=1+ # drains it to one entry and the 7-day idle timer takes the last.+ pre=$(printf '%s' "$key" | sed -E 's/-[0-9a-f]{64}$//')+ # fe-dist has no restore-keys: an older generation is the only+ # thing an exact revert of the frontend can hit, so it keeps `keep`.+ case "$key" in fe-dist-*) ;; *) fkeep=1 ;; esac ;;*) continue ;;esac# Unshortened means the suffix did not match, which would make@@ -381,7 +380,7 @@ jobs:echo "| metric | value |"echo "| --- | --- |"echo "| mode | $([ "$DELETE" = yes ] && echo delete || echo 'report only') |"- echo "| kept per prefix | $KEEP |"+ echo "| kept per prefix | 1 (fe-dist: $KEEP) |"echo "| caches total | $count |"echo "| dead-family candidates | $dead |"echo "| closed-PR candidates | $stale_pr |"@@ -396,5 +395,5 @@ jobs:# 80% of the 50 GiB ceiling. Above this a single dependency bump can# push the repo over between two scheduled sweeps.if [ "$after" -gt 42949672960 ]; then- echo "::warning::Cache usage is $(gib "$after") GiB of 50 GiB after pruning. Lower keep to 1, or find the family that grew."+ echo "::warning::Cache usage is $(gib "$after") GiB of 50 GiB after pruning. Find the family that grew."fidiff --git a/.github/workflows/consolidated-tests-ci.yml b/.github/workflows/consolidated-tests-ci.ymlindex 1d5ac4dc0d6..1dcb6be7ec7 100644--- a/.github/workflows/consolidated-tests-ci.yml+++ b/.github/workflows/consolidated-tests-ci.yml@@ -2364,19 +2364,16 @@ jobs:- name: Restore the pip cacheid: pip-cache- # Its own `name`, not `consolidated`'s. Two jobs sharing one is the- # defect tests/studio/test_pip_cache_naming.py exists to reject: the- # save is gated on `cache-hit != 'true'`, so whichever job finished- # first on main would write the entry and the other would restore it- # and never save its own. The key-files are the same because the- # install is the same, which that file also allows for explicitly.- #- # The first run on a pull request therefore misses and downloads. The- # entry is written by the next push to main and warm from then on, the- # same lifecycle every other key here has.+ # `consolidated`'s entry, read-only: this job has no pip-cache-save. The+ # install is the same core-cpu-setup call on the same matrix (asserted by+ # tests/studio/test_zoo_suite_parallel_isolation.py), so a separate entry+ # held the same ~1.4 GiB of wheels twice. A shared name is only a defect+ # when both jobs save, since then the first to finish wins and the other's+ # extras are never written; with one writer there is nothing to lose.+ # tests/studio/test_pip_cache_naming.py allows exactly this borrower.uses: ./.github/actions/pip-cache-restorewith:- name: consolidated-zoo+ name: consolidatedkey-files: |pyproject.tomlstudio/backend/requirements/*.txt@@ -2513,14 +2510,6 @@ jobs:echo " - unsloth_zoo @ ${UNSLOTH_ZOO_REF} pytest tests/ (GPU-only cases deselected)"echo " - unsloth_zoo.compiler.test_apply_fused_lm_head"- - name: Save the pip cache- if: always()- uses: ./.github/actions/pip-cache-save- with:- dir: ${{ steps.pip-cache.outputs.dir }}- key: ${{ steps.pip-cache.outputs.key }}- cache-hit: ${{ steps.pip-cache.outputs.cache-hit }}-llama-cpp-smoke:# Standalone llama.cpp build + smoke. Earlier this lived inside every# consolidated matrix cell and re-cmake'd llama.cpp ~5 min per cell --diff --git a/tests/studio/test_cache_budget_discipline.py b/tests/studio/test_cache_budget_discipline.pyindex e52b7b4dca3..4a833637261 100644--- a/tests/studio/test_cache_budget_discipline.py+++ b/tests/studio/test_cache_budget_discipline.py@@ -17,8 +17,8 @@ WORKFLOWS = REPO / ".github" / "workflows"# ungated on PR refs and evicts main's shared entry.PIP_CACHE_JOBS = {("consolidated-tests-ci.yml", "consolidated"),- # Same install as `consolidated` but needs its own key: saves are gated on- # `cache-hit != 'true'`, so a shared key lets only one job ever save.+ # Same install as `consolidated`, so it restores that entry read-only and never+ # saves; see READ_ONLY_BORROWERS in test_pip_cache_naming.py.("consolidated-tests-ci.yml", "consolidated-zoo"),("consolidated-tests-ci.yml", "llama-cpp-smoke"),("mlx-ci.yml", "dispatch"),@@ -35,6 +35,10 @@ PIP_CACHE_JOBS = {# Partial saves on purpose: ccache checksums entries, so a truncated cache only costs misses.PARTIAL_SAVE_JOBS = {("prebuilt-cuda-wheels.yml", "warm")}+# Restore another job's pip cache and deliberately save nothing (mirrors+# READ_ONLY_BORROWERS in test_pip_cache_naming.py, which checks the owner side).+PIP_CACHE_READ_ONLY_JOBS = {("consolidated-tests-ci.yml", "consolidated-zoo")}+HEAVY = re.compile(r"torch|transformers|trl|peft|vllm|bitsandbytes|sentence-transformers|diffusers"r"|accelerate|datasets|requirements/"@@ -408,6 +412,11 @@ def test_every_pip_cache_scopes_its_key_to_what_it_installs(name, jid):def test_every_restored_pip_cache_is_also_saved_and_wired_to_its_restore(name, jid):"""A restore without a save fills nothing; a save reading the wrong restore outputs writes nothing."""restore, save = _pip_cache_steps(name, jid)+ if (name, jid) in PIP_CACHE_READ_ONLY_JOBS:+ assert (+ restore is not None and save is None+ ), f"{name}:{jid} is a read-only borrower: it must restore and must not save"+ returnassert restore is not None and save is not None, (f"{name}:{jid} has restore={restore is not None}, save={save is not None}; the "f"pair has to stay together or the cache is never populated"diff --git a/tests/studio/test_pip_cache_naming.py b/tests/studio/test_pip_cache_naming.pyindex 4584a95c7c8..6502e53cbcc 100644--- a/tests/studio/test_pip_cache_naming.py+++ b/tests/studio/test_pip_cache_naming.py@@ -15,6 +15,11 @@ WORKFLOWS = sorted((REPO / ".github" / "workflows").glob("*.yml"))RESTORE_SUFFIX = "actions/pip-cache-restore"SAVE_SUFFIX = "actions/pip-cache-save"+# Jobs that restore another job's entry and never save. Only safe when the install+# is identical, so a borrower's extras cannot exist; consolidated-zoo runs the same+# core-cpu-setup on the same matrix, asserted by test_zoo_suite_parallel_isolation.py.+READ_ONLY_BORROWERS = {"consolidated-tests-ci.yml:consolidated-zoo": "consolidated"}+def _jobs(path):doc = yaml.safe_load(path.read_text(encoding = "utf-8")) or {}@@ -58,6 +63,14 @@ def test_every_restore_names_itself(path):assert files, f"{path.name}:{job} passes no key-files"+def _saves(steps):+ return [s for s in steps if str(s.get("uses", "")).endswith(SAVE_SUFFIX)]+++def _key_files(steps):+ return " ".join(((_restores(steps)[0].get("with") or {}).get("key-files") or "").split())++def test_no_two_jobs_share_a_cache_name():seen = {}for path in WORKFLOWS:@@ -65,6 +78,8 @@ def test_no_two_jobs_share_a_cache_name():for step in _restores(steps):name = (step.get("with") or {}).get("name") or ""where = f"{path.name}:{job}"+ if READ_ONLY_BORROWERS.get(where) == name:+ continueassert name not in seen, (f"cache name {name!r} is used by both {seen[name]} and {where}. "f"A shared name is a shared key: only the first job to finish on "@@ -77,10 +92,13 @@ def test_jobs_sharing_key_files_still_have_distinct_names():by_files = {}for path in WORKFLOWS:for job, steps in _jobs(path):+ where = f"{path.name}:{job}"for step in _restores(steps):with_ = step.get("with") or {}+ if READ_ONLY_BORROWERS.get(where) == with_.get("name"):+ continuefiles = " ".join((with_.get("key-files") or "").split())- by_files.setdefault(files, []).append((f"{path.name}:{job}", with_.get("name")))+ by_files.setdefault(files, []).append((where, with_.get("name")))for files, entries in by_files.items():names = [n for _, n in entries]assert len(names) == len(set(names)), (@@ -89,6 +107,32 @@ def test_jobs_sharing_key_files_still_have_distinct_names():)[email protected]("where,owner", sorted(READ_ONLY_BORROWERS.items()))+def test_read_only_borrowers_never_save_and_their_owner_does(where, owner):+ jobs = {f"{p.name}:{j}": steps for p in WORKFLOWS for j, steps in _jobs(p)}+ assert where in jobs, f"{where} no longer exists; drop it from READ_ONLY_BORROWERS"+ restores = _restores(jobs[where])+ assert [(s.get("with") or {}).get("name") for s in restores] == [+ owner+ ], f"{where} must restore exactly one pip cache, {owner!r}'s"+ assert not _saves(jobs[where]), (+ f"{where} borrows {owner!r}'s entry, so it must not save: two writers of one key "+ f"race, and the loser's wheels are never written"+ )+ owners = [+ w+ for w, steps in jobs.items()+ if w != where+ and _saves(steps)+ and any((s.get("with") or {}).get("name") == owner for s in _restores(steps))+ ]+ assert len(owners) == 1, f"{owner!r} needs exactly one saving job, found {owners}"+ assert _key_files(jobs[where]) == _key_files(jobs[owners[0]]), (+ f"{where} keys on different files than {owners[0]}, so it would look up a key "+ f"its owner never writes"+ )++def test_key_is_versioned_and_carries_the_name():action = (REPO / ".github/actions/pip-cache-restore/action.yml").read_text(encoding = "utf-8")assert 'prefix="pip-v2-${name}-' in action, (diff --git a/tests/studio/test_zoo_suite_parallel_isolation.py b/tests/studio/test_zoo_suite_parallel_isolation.pyindex 621cd234779..c482b1b90df 100644--- a/tests/studio/test_zoo_suite_parallel_isolation.py+++ b/tests/studio/test_zoo_suite_parallel_isolation.py@@ -286,11 +286,8 @@ def test_both_halves_of_core_share_one_install_preamble() -> None:f"same environment, so their preambles have to be the same steps in the same order")for left, right in zip(a, b):- # A shared pip cache name is a shared key; see tests/studio/test_pip_cache_naming.py.- left, right = dict(left), dict(right)- if "pip-cache-restore" in str(left.get("uses", "")):- left["with"] = {k: v for k, v in left["with"].items() if k != "name"}- right["with"] = {k: v for k, v in right["with"].items() if k != "name"}+ # The pip cache restore is compared whole, name included: consolidated-zoo+ # borrows consolidated's entry read-only (see test_pip_cache_naming.py).assert left == right, (f"the Core preambles have drifted at step "f"{left.get('name') or left.get('uses')!r}:\n consolidated: {left}\n"