Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.
Python agentaichatgptdeepseekfine-tuninggemmaimage-generationllamallmllmsopenaipythonqwenreinforcement-learningself-hostedstable-diffusiontext-to-speechttsuiunsloth
CI: coalesce main-push runs per workflow so merge bursts stop outrunning the runner pool (#13705)
Daniel Han80157cacb4e5
diff --git a/.github/workflows/clean-machine-install-ci.yml b/.github/workflows/clean-machine-install-ci.ymlindex 5c52c2905a3..b0f0c2c2a2a 100644--- a/.github/workflows/clean-machine-install-ci.yml+++ b/.github/workflows/clean-machine-install-ci.yml@@ -158,11 +158,10 @@ concurrency:# The event is in the group so a scheduled run and a push to main, which share a ref,# cannot coalesce: only one run stays pending per group, and cancel-in-progress protects# the running run, never the pending one, so a merge burst would drop the nightly.- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/consolidated-tests-ci.yml b/.github/workflows/consolidated-tests-ci.ymlindex d4bcb31a253..e0cf0045f32 100644--- a/.github/workflows/consolidated-tests-ci.yml+++ b/.github/workflows/consolidated-tests-ci.yml@@ -126,11 +126,10 @@ on:default: 'main'concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/cross-platform-parity-ci.yml b/.github/workflows/cross-platform-parity-ci.ymlindex 0bed5bbec8d..82ff2027602 100644--- a/.github/workflows/cross-platform-parity-ci.yml+++ b/.github/workflows/cross-platform-parity-ci.yml@@ -280,11 +280,10 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/desktop-app-clean-machine-ci.yml b/.github/workflows/desktop-app-clean-machine-ci.ymlindex a1bd8628655..3b03aecdc74 100644--- a/.github/workflows/desktop-app-clean-machine-ci.yml+++ b/.github/workflows/desktop-app-clean-machine-ci.yml@@ -85,10 +85,9 @@ on:concurrency:group: ${{ github.workflow }}-${{ github.ref }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/interrupted-install-ci.yml b/.github/workflows/interrupted-install-ci.ymlindex a6467d21f14..b314c917824 100644--- a/.github/workflows/interrupted-install-ci.yml+++ b/.github/workflows/interrupted-install-ci.yml@@ -94,10 +94,10 @@ on:concurrency:# The event is in the group so a scheduled run and a push to main, which share a ref,- # cannot cancel each other; the sha makes it unique per commit on main so a merge burst- # cannot cancel a pending run (see studio-backend-ci.yml). On a PR branch the group is- # the ref, so a new push cancels the run for the previous one.- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}+ # cannot replace each other. On main a merge burst runs the executing commit plus the+ # tip (see studio-backend-ci.yml). On a PR branch the group is the ref, so a new push+ # cancels the run for the previous one.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/lint-ci.yml b/.github/workflows/lint-ci.ymlindex 6c8886aab09..69056019807 100644--- a/.github/workflows/lint-ci.yml+++ b/.github/workflows/lint-ci.yml@@ -35,11 +35,10 @@ on:branches: [main]concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}permissions:diff --git a/.github/workflows/local-agent-guides-ci.yml b/.github/workflows/local-agent-guides-ci.ymlindex 3695a010bf9..2d383c5c28a 100644--- a/.github/workflows/local-agent-guides-ci.yml+++ b/.github/workflows/local-agent-guides-ci.yml@@ -134,11 +134,10 @@ concurrency:# coalesce. Both resolve to refs/heads/main, and the default queue: single keeps# only one pending run, so without this a merge burst silently drops the nightly:# cancel-in-progress protects the running run, never the pending one.- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/lockfile-audit.yml b/.github/workflows/lockfile-audit.ymlindex 4b68b357128..9cea7cce62c 100644--- a/.github/workflows/lockfile-audit.yml+++ b/.github/workflows/lockfile-audit.yml@@ -42,11 +42,10 @@ concurrency:# coalesce. Both resolve to refs/heads/main, and the default queue: single keeps# only one pending run, so without this a merge burst silently drops the nightly:# cancel-in-progress protects the running run, never the pending one.- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}permissions:diff --git a/.github/workflows/mlx-ci.yml b/.github/workflows/mlx-ci.ymlindex 604091d3914..d1d9ec0915c 100644--- a/.github/workflows/mlx-ci.yml+++ b/.github/workflows/mlx-ci.yml@@ -180,11 +180,10 @@ on:- '.github/actions/pip-cache-save/**'concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}permissions:diff --git a/.github/workflows/notebooks-ci.yml b/.github/workflows/notebooks-ci.ymlindex bd7e996d3e8..6cd276add61 100644--- a/.github/workflows/notebooks-ci.yml+++ b/.github/workflows/notebooks-ci.yml@@ -70,10 +70,9 @@ on:concurrency:group: ${{ github.workflow }}-${{ github.ref }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/security-audit.yml b/.github/workflows/security-audit.ymlindex 6caa7ed0d17..a4872638c2e 100644--- a/.github/workflows/security-audit.yml+++ b/.github/workflows/security-audit.yml@@ -103,11 +103,10 @@ concurrency:# coalesce. Both resolve to refs/heads/main, and the default queue: single keeps# only one pending run, so without this a merge burst silently drops the nightly:# cancel-in-progress protects the running run, never the pending one.- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/studio-api-smoke.yml b/.github/workflows/studio-api-smoke.ymlindex b831fea4e77..3ff82a32913 100644--- a/.github/workflows/studio-api-smoke.yml+++ b/.github/workflows/studio-api-smoke.yml@@ -88,11 +88,10 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/studio-backend-ci.yml b/.github/workflows/studio-backend-ci.ymlindex 8b719f36281..2342f45bd06 100644--- a/.github/workflows/studio-backend-ci.yml+++ b/.github/workflows/studio-backend-ci.yml@@ -101,9 +101,17 @@ on:- '.github/workflows/security-audit.yml'concurrency:- # Unique per commit on main, so a merge burst cannot cancel a pending run- # before it starts. See the note below for what this is buying.- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}+ # One group per workflow, ref and event. On a PR branch a new push cancels the old run.+ # On main cancel-in-progress is false, so a RUNNING run always finishes, and GitHub keeps+ # one PENDING run per group, replaced by each newer push: a merge burst runs the commit+ # already executing plus the tip, never a commit in between. That is the intended trade.+ # A group per commit ran every workflow on every main commit (about 550 runner-minutes+ # each at 100-160 commits a day), which is more than the org's runner pool supplies, so+ # every queue grew for hours. Commits in the middle of a burst were tested as PRs, the+ # tip's tree contains them, and a red tip is bisected with workflow_dispatch. The event+ # keeps a nightly or a dispatch from being replaced by a push. The contract is enforced+ # by tests/studio/test_main_runs_survive_merge_bursts.py.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}permissions:diff --git a/.github/workflows/studio-export-capability-ci.yml b/.github/workflows/studio-export-capability-ci.ymlindex 38be0af8fd4..1a5041794f0 100644--- a/.github/workflows/studio-export-capability-ci.yml+++ b/.github/workflows/studio-export-capability-ci.yml@@ -74,11 +74,10 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/studio-frontend-ci.yml b/.github/workflows/studio-frontend-ci.ymlindex e8033811bd9..9bbd23a2c12 100644--- a/.github/workflows/studio-frontend-ci.yml+++ b/.github/workflows/studio-frontend-ci.yml@@ -93,11 +93,10 @@ on:- '.github/scripts/retry-with-apt-lock.sh'concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}permissions:diff --git a/.github/workflows/studio-inference-smoke.yml b/.github/workflows/studio-inference-smoke.ymlindex 02548d54cbd..b1521c5d2ca 100644--- a/.github/workflows/studio-inference-smoke.yml+++ b/.github/workflows/studio-inference-smoke.yml@@ -104,11 +104,10 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/studio-load-orchestrator-ci.yml b/.github/workflows/studio-load-orchestrator-ci.ymlindex 23e924f82aa..8141cd27a87 100644--- a/.github/workflows/studio-load-orchestrator-ci.yml+++ b/.github/workflows/studio-load-orchestrator-ci.yml@@ -28,11 +28,10 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}permissions:diff --git a/.github/workflows/studio-mac-install-matrix.yml b/.github/workflows/studio-mac-install-matrix.ymlindex 03b724707db..0df0d7d46a8 100644--- a/.github/workflows/studio-mac-install-matrix.yml+++ b/.github/workflows/studio-mac-install-matrix.yml@@ -109,11 +109,10 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/studio-mac-ui-smoke.yml b/.github/workflows/studio-mac-ui-smoke.ymlindex 01a0df2bd3e..d3913b76b8e 100644--- a/.github/workflows/studio-mac-ui-smoke.yml+++ b/.github/workflows/studio-mac-ui-smoke.yml@@ -140,11 +140,10 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/studio-tauri-smoke.yml b/.github/workflows/studio-tauri-smoke.ymlindex 11bd45e9988..5c0640a713f 100644--- a/.github/workflows/studio-tauri-smoke.yml+++ b/.github/workflows/studio-tauri-smoke.yml@@ -70,11 +70,10 @@ on:- 'studio/backend/tests/fixtures/access_log_records.json'concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}permissions:diff --git a/.github/workflows/studio-ui-smoke.yml b/.github/workflows/studio-ui-smoke.ymlindex e84d9ccf635..f6a303c3dc5 100644--- a/.github/workflows/studio-ui-smoke.yml+++ b/.github/workflows/studio-ui-smoke.yml@@ -119,9 +119,9 @@ on:workflow_dispatch:concurrency:- # Unique per commit on main, so a merge burst cannot cancel a pending run- # before it starts. See the note below for what this is buying.- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}+ # Per workflow, ref and event: a merge burst runs the executing commit plus the tip.+ # See studio-backend-ci.yml for the trade, and the note below.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}# On a PR branch: latest-only, which is what cancel-in-progress buys.## On main it buys less than it looks like. GitHub cancels any PENDING run in adiff --git a/.github/workflows/studio-update-smoke.yml b/.github/workflows/studio-update-smoke.ymlindex fe0f14bd21f..549163e1318 100644--- a/.github/workflows/studio-update-smoke.yml+++ b/.github/workflows/studio-update-smoke.yml@@ -163,11 +163,10 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/studio-vae-seam-guard.yml b/.github/workflows/studio-vae-seam-guard.ymlindex 6d099f86e50..4c0df267743 100644--- a/.github/workflows/studio-vae-seam-guard.yml+++ b/.github/workflows/studio-vae-seam-guard.yml@@ -34,8 +34,8 @@ permissions:contents: readconcurrency:- # Unique per commit on main, so a merge burst cannot cancel a main run; superseded PR pushes are cancelled.- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}+ # Superseded PR pushes are cancelled; on main a running run finishes and a burst runs the tip (see studio-backend-ci.yml).+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}cancel-in-progress: ${{ github.event_name == 'pull_request' }}jobs:diff --git a/.github/workflows/studio-windows-api-smoke.yml b/.github/workflows/studio-windows-api-smoke.ymlindex a3f509ba659..d2ac5164dde 100644--- a/.github/workflows/studio-windows-api-smoke.yml+++ b/.github/workflows/studio-windows-api-smoke.yml@@ -77,11 +77,10 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/studio-windows-inference-smoke.yml b/.github/workflows/studio-windows-inference-smoke.ymlindex 85fa9f475f9..f2f87bc3f0c 100644--- a/.github/workflows/studio-windows-inference-smoke.yml+++ b/.github/workflows/studio-windows-inference-smoke.yml@@ -99,11 +99,10 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/studio-windows-ui-smoke.yml b/.github/workflows/studio-windows-ui-smoke.ymlindex efcc9b88a36..aee85fe4b8e 100644--- a/.github/workflows/studio-windows-ui-smoke.yml+++ b/.github/workflows/studio-windows-ui-smoke.yml@@ -89,11 +89,10 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/studio-windows-update-smoke.yml b/.github/workflows/studio-windows-update-smoke.ymlindex 180621726a1..4e075783f74 100644--- a/.github/workflows/studio-windows-update-smoke.yml+++ b/.github/workflows/studio-windows-update-smoke.yml@@ -85,11 +85,10 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/studiobench-ci.yml b/.github/workflows/studiobench-ci.ymlindex 29979781d47..34345f8e959 100644--- a/.github/workflows/studiobench-ci.yml+++ b/.github/workflows/studiobench-ci.yml@@ -111,9 +111,9 @@ on:workflow_dispatch:concurrency:- # Unique per commit on main, so a merge burst cannot cancel a pending run- # before it starts. Matches studio-ui-smoke.yml, for the reasons documented there.- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}+ # Per workflow, ref and event: a merge burst runs the executing commit plus the tip.+ # Matches studio-backend-ci.yml, for the reasons documented there.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/studiobench-ui-parity.yml b/.github/workflows/studiobench-ui-parity.ymlindex 18e76800461..50e3fc5d43d 100644--- a/.github/workflows/studiobench-ui-parity.yml+++ b/.github/workflows/studiobench-ui-parity.yml@@ -200,7 +200,7 @@ on:# this workflow reads that entry rather than minting a second copy of the same# bytes against a cache budget measured 99.3% full.concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}permissions:diff --git a/.github/workflows/version-compat-ci.yml b/.github/workflows/version-compat-ci.ymlindex 803e8c43d90..5bc577c26dc 100644--- a/.github/workflows/version-compat-ci.yml+++ b/.github/workflows/version-compat-ci.yml@@ -39,10 +39,9 @@ on:concurrency:group: ${{ github.workflow }}-${{ github.ref }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/wheel-smoke.yml b/.github/workflows/wheel-smoke.ymlindex e32588b3cab..f709cb4a26d 100644--- a/.github/workflows/wheel-smoke.yml+++ b/.github/workflows/wheel-smoke.yml@@ -48,11 +48,10 @@ on:- 'scripts/lockfile_supply_chain_audit.py'concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}- # Latest-only on a PR branch. On main this does less than it reads like: it stops- # a RUNNING main job being killed, but GitHub cancels any PENDING run in the group- # the moment a newer one is queued, so a merge burst still leaves only the tip.- # See studio-backend-ci.yml, which is grouped per commit on main for that reason.+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}+ # Latest-only on a PR branch. On main, cancel-in-progress is false so a RUNNING run is+ # never killed, and a newer push replaces only the PENDING one: a merge burst runs the+ # commit already executing plus the tip. See studio-backend-ci.yml for the trade.cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}permissions:diff --git a/.github/workflows/windows-application-control-ci.yml b/.github/workflows/windows-application-control-ci.ymlindex ac69babced9..3d24549df92 100644--- a/.github/workflows/windows-application-control-ci.yml+++ b/.github/workflows/windows-application-control-ci.yml@@ -89,7 +89,7 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/windows-arm64-ci.yml b/.github/workflows/windows-arm64-ci.ymlindex 957fb73523c..968cf553499 100644--- a/.github/workflows/windows-arm64-ci.yml+++ b/.github/workflows/windows-arm64-ci.yml@@ -83,7 +83,7 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/windows-no-compiler-ci.yml b/.github/workflows/windows-no-compiler-ci.ymlindex 88f94088da3..3968d59e185 100644--- a/.github/workflows/windows-no-compiler-ci.yml+++ b/.github/workflows/windows-no-compiler-ci.yml@@ -106,7 +106,7 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/.github/workflows/workflow-trigger-lint.yml b/.github/workflows/workflow-trigger-lint.ymlindex e7a5a106d57..64a69abcf61 100644--- a/.github/workflows/workflow-trigger-lint.yml+++ b/.github/workflows/workflow-trigger-lint.yml@@ -30,7 +30,7 @@ on:workflow_dispatch:concurrency:- group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}+ group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}cancel-in-progress: ${{ github.event_name == 'pull_request' }}permissions:diff --git a/tests/studio/test_main_runs_survive_merge_bursts.py b/tests/studio/test_main_runs_survive_merge_bursts.pyindex f06bd7f5ec5..748a6306a01 100644--- a/tests/studio/test_main_runs_survive_merge_bursts.py+++ b/tests/studio/test_main_runs_survive_merge_bursts.py@@ -1,22 +1,34 @@# SPDX-License-Identifier: AGPL-3.0-only# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0-"""``cancel-in-progress: false`` does not keep a main run alive, and the repo believed it did.--GitHub cancels any PENDING run in a concurrency group the moment a newer one is queued.-``cancel-in-progress`` governs only runs that are already EXECUTING: "any existing pending-job or workflow in the same concurrency group will be canceled and the new queued job or-workflow will take its place" (workflow-syntax reference).--So on a branch where pushes land in bursts, ``cancel-in-progress: false`` buys nothing for-the runs in the middle of the burst. Twenty-seven workflows carried a comment saying it did.-Both incidents the repo wrote down are this: studio-ui-smoke.yml records a break sitting on-main "for 14 hours behind four cancelled runs", and on 2026-08-17 five merges cancelled four-consecutive main runs of Backend CI, which completed none. Each was cancelled with ZERO jobs-recorded, so nothing ever showed up as a failure; the workflow simply never reported.--The only thing that protects a main run is a concurrency group it does not share with the-next commit.+"""A merge burst on main runs the tip, never kills a running run, and nothing more.++How GitHub treats one concurrency group: at most one run EXECUTES and at most one is+PENDING. A newer run replaces the pending one ("any existing pending job or workflow in the+same concurrency group will be canceled and the new queued job or workflow will take its+place", workflow-syntax reference), and ``cancel-in-progress`` decides only whether the+EXECUTING run is killed as well.++This repo used to give every main commit its own group (``github.sha`` in the group on+main), so no main run ever replaced another. That bought a verdict per commit and cost+every workflow on every commit: about 550 runner-minutes per main commit at 100-160+commits a day, more than the org's runner pool supplies, so every queue (PRs, zoo, the+cache janitor) waited hours behind main.++The contract now, for every workflow that runs on pushes to main:++ 1. One group per workflow, ref and event. A merge burst on main leaves the run already+ executing plus ONE pending run, which a newer push keeps replacing, so the tip always+ runs. The commits in between were tested as pull requests and the tip's tree contains+ them; a red tip is bisected with workflow_dispatch. That lost per-commit attribution on+ main is the trade, and it is deliberate.+ 2. ``cancel-in-progress`` is false on main, so a run that has started always finishes+ (it is the one saving the main-branch caches, among other things).+ 3. The event is in the group, so a nightly or a dispatch on main is never replaced by a+ push, and the reverse.+ 4. On a pull request two pushes share a group, so the newer push supersedes the older.++Per-commit groups remain only where named below, with the reason."""import re@@ -30,9 +42,17 @@ WORKFLOWS = REPO / ".github" / "workflows"# Kaggle spends an external GPU quota rather than runner minutes.QUOTA_BOUND = frozenset({"kaggle-t4-notebook-ci.yml", "kaggle-t4-studio-gpu-ci.yml"})-# Phrasings that assert main runs are not cancelled.+# Allowed to keep a group per commit on main. codeql.yml analyses each main commit for code+# scanning alerts and manages its own cache retention; it may coalesce too, so it is exempt+# from the rule rather than required to break it.+PER_COMMIT_ALLOWED = frozenset({"codeql.yml"})++# Phrasings that assert a main run is never cancelled or that main is grouped per commit.+# Either is false for a workflow sharing one group across main commits: its PENDING run is+# replaced by the next push.CLAIMS_PROTECTION = re.compile(- r"never on main|never cancelled on main|not cancelled on main|never cancels on main",+ r"never on main|never cancelled on main|not cancelled on main|never cancels on main"+ r"|unique per commit on main|grouped per commit on main|one group per commit on main",re.IGNORECASE,)@@ -96,6 +116,8 @@ def _evaluate(expression: str, context: dict[str, str]) -> str:if ternary:taken = ternary.group(2) if _condition(ternary.group(1), context) else ternary.group(3)return _term(taken, context)+ if _COMPARISON.fullmatch(expression.strip()):+ return "true" if _condition(expression, context) else "false"return _term(expression, context)@@ -145,31 +167,83 @@ def _protected() -> dict[str, dict]:}-def test_every_workflow_that_runs_on_main_is_grouped_per_commit():- """Otherwise a merge burst discards it before it starts.+def _cancel_setting(document: dict) -> str:+ concurrency = document.get("concurrency")+ if isinstance(concurrency, dict):+ return str(concurrency.get("cancel-in-progress", False)).lower()+ return "false"+++def _cancels_on_main_push(document: dict) -> bool:+ rendered = _render(_cancel_setting(document), ref = MAIN, sha = "a" * 40, event_name = "push")+ return rendered.strip().lower() == "true"+- Checked across all of them rather than a named few, because the failure is invisible:- the run is cancelled with zero jobs recorded, so nothing surfaces as a test failure and- the workflow simply never reported on that commit.+def _coalescing() -> dict[str, dict]:+ return {name: doc for name, doc in _protected().items() if name not in PER_COMMIT_ALLOWED}+++def test_every_workflow_that_runs_on_main_coalesces_a_merge_burst():+ """Two main commits share one group, so a burst runs the executing commit plus the tip.++ A group per commit runs every workflow on every main commit, which is the load that+ queued the whole org for hours. Checked across all of them rather than a named few."""offenders = sorted(- name+ name for name, document in _coalescing().items() if _is_per_commit_on_main(_group(document))+ )+ assert not offenders, (+ f"{offenders} give every main commit its own concurrency group, so each main commit "+ f"runs the whole workflow and nothing is ever superseded. Use "+ f"${{{{ github.workflow }}}}-${{{{ github.ref }}}}-${{{{ github.event_name }}}} with "+ f"cancel-in-progress false on main, or add the file to PER_COMMIT_ALLOWED with a reason."+ )+++def test_a_running_main_run_is_never_cancelled():+ """Coalescing replaces the PENDING run only; the one executing must finish."""+ offenders = {+ name: _cancel_setting(document)for name, document in _protected().items()- if not _is_per_commit_on_main(_group(document))+ if _cancels_on_main_push(document)+ }+ assert not offenders, (+ f"{offenders} cancel a running main run when the next commit lands, so a merge burst "+ f"can leave main with no finished run at all. Gate it on github.event_name == 'pull_request'.")+++def test_a_push_never_replaces_a_nightly_or_a_dispatch_on_main():+ """The event is in the group, so runs of different kinds on main keep their own slot."""+ offenders = {}+ for name, document in _coalescing().items():+ triggers = document.get(True) or document.get("on") or {}+ group = _group(document)+ push = _render(group, ref = MAIN, sha = "a" * 40, event_name = "push")+ for event in ("schedule", "workflow_dispatch", "repository_dispatch"):+ if (+ event in triggers+ and _render(group, ref = MAIN, sha = "a" * 40, event_name = event) == push+ ):+ offenders.setdefault(name, []).append(event)assert not offenders, (- f"{offenders} share one concurrency group across main commits, so a merge burst "- f"cancels their pending runs before they start. cancel-in-progress does not prevent "- f"that: it only spares runs already executing. Append "- f"-${{{{ github.ref == 'refs/heads/main' && github.sha || '' }}}} to the group."+ f"{offenders} share the push group on main with another trigger, so a merge burst "+ f"replaces a pending nightly or dispatch. Put ${{{{ github.event_name }}}} in the group.")+def test_the_per_commit_allowance_still_names_workflows_that_exist():+ documents = _documents()+ missing = sorted(name for name in PER_COMMIT_ALLOWED if name not in documents)+ assert not missing, f"PER_COMMIT_ALLOWED names workflows that no longer exist: {missing}"++def test_the_scan_actually_found_the_workflows():"""A glob that matched nothing would pass every check above."""protected = _protected()assert len(protected) > 20, f"only found {len(protected)} main workflows; the scan is wrong"assert "studio-backend-ci.yml" in protected+ assert len(_coalescing()) > 20, "the coalescing scan found almost nothing; the scan is wrong"def test_this_guard_runs_on_a_workflow_only_pull_request():@@ -241,11 +315,11 @@ def test_every_group_expression_is_understood():def test_the_evaluator_reads_which_branch_supplies_the_sha():- """The predicate above is only worth its assertions if this holds.+ """The per-commit predicate is only worth its assertions if this holds.- Each of these renders to a string containing ``github.sha`` in its source text, and the- substring test that this file first shipped with called all four per-commit. Only the- first one is.+ Each of these contains ``github.sha`` in its source text, and a substring test would+ call all four per-commit. Only the first and the last are, so a grep cannot tell a+ coalescing group from a per-commit one; rendering can."""gated = "${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}"reversed_ = "${{ github.ref }}-${{ github.ref != 'refs/heads/main' && github.sha || '' }}"@@ -268,7 +342,7 @@ def test_the_evaluator_reads_which_branch_supplies_the_sha():def test_no_workflow_claims_a_main_protection_it_does_not_have():- """The comment that made this repo believe it was covered, in 27 files.+ """A comment promising per-commit or never-cancelled main runs must match the group.Read from the raw text rather than the parsed document, because a comment is exactly whatYAML throws away, and the comment is what people act on.@@ -285,7 +359,7 @@ def test_no_workflow_claims_a_main_protection_it_does_not_have():if isinstance(document, dict) and not _is_per_commit_on_main(_group(document)):offenders.append(path.name)assert not offenders, (- f"{offenders} say main runs are never cancelled while sharing one concurrency group "- f"across main commits, which is the belief that let a regression sit on main for 14 "- f"hours behind four cancelled runs. Either group per commit, or do not claim it."+ f"{offenders} claim main runs are per commit or never cancelled while sharing one "+ f"concurrency group across main commits, where a newer push replaces the pending run. "+ f"Describe what the group does (see studio-backend-ci.yml).")