Skip to content

Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.

Say when a failing pull_request job is testing a stale merge ref (#11386)

Daniel Han8090d9523fe0
diff --git a/.github/actions/merge-ref-age/action.yml b/.github/actions/merge-ref-age/action.yml
new file mode 100644
index 00000000000..acf9d9db735
--- /dev/null
+++ b/.github/actions/merge-ref-age/action.yml
@@ -0,0 +1,68 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+# Say, on a failure, how far behind the base branch this run's merge ref is.
+#
+# A pull_request run does not test the PR's head. It tests `refs/pull/N/merge`, a
+# merge of the head into the base as the base stood WHEN THE RUN WAS CREATED. That
+# ref is not recomputed because the base later moved, so a failure fixed on main
+# ten minutes after a run started keeps failing that run forever, and re-running it
+# does not help: a re-run replays the same merge commit.
+#
+# That is not hypothetical here. #11372 gave three test files `encoding = "utf-8"`
+# and merged at 14:35 UTC. Runs created at 13:39, 14:26 and 14:29 on #11375, #11357
+# and #11373 all kept failing `tests/test_source_read_encoding.py` on exactly those
+# three lines, and all three were read as defects in branches that had never touched
+# the files. The only cure is a push to the PR, which is what makes GitHub build a
+# new merge ref.
+#
+# The cost of not saying this is that someone debugs a fix that already landed. The
+# step is `if: failure()` so a green job pays nothing for it, and it only speaks when
+# the base has actually moved, so it does not add noise to the common case where the
+# failure is real.
+#
+# It NEVER changes the job's result. A stale merge ref does not make a failure
+# untrue -- the merge really does fail -- and deciding that for the reader would be
+# worse than the confusion it is fixing. It reports and gets out of the way.
+
+name: Report the age of this run's merge ref
+description: >-
+ On a pull_request run, emit a notice saying how many commits the base branch
+ has gained since this run's merge ref was built. Advisory only: it never fails,
+ and never changes the outcome of the job it sits in.
+
+runs:
+ using: composite
+ steps:
+ - name: Compare the merge ref's base against the branch tip
+ shell: bash
+ env:
+ # `gh` needs a token, and checkout runs with persist-credentials: false
+ # in these jobs, so there is nothing in the git config to fall back on.
+ # The compare endpoint only wants `contents: read`.
+ GH_TOKEN: ${{ github.token }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ BASE_REF: ${{ github.event.pull_request.base.ref }}
+ REPOSITORY: ${{ github.repository }}
+ run: |
+ set -u
+ if [ -z "${BASE_SHA:-}" ] || [ -z "${BASE_REF:-}" ]; then
+ # Not a pull_request event. Nothing to compare against.
+ exit 0
+ fi
+ # base.sha is the commit the merge ref was built on; BASE_REF is where the
+ # branch is now. `ahead_by` on base...ref is how far the branch has moved
+ # since, which is the number this is about. Asking the API rather than the
+ # clone on purpose: these checkouts are shallow, and deepening them enough
+ # to hold both commits costs more than the answer is worth.
+ behind=$(gh api "repos/${REPOSITORY}/compare/${BASE_SHA}...${BASE_REF}" --jq '.ahead_by' 2>/dev/null) || behind=""
+ case "${behind}" in
+ ''|*[!0-9]*)
+ # No answer, or not a number. Say nothing rather than guess.
+ exit 0
+ ;;
+ esac
+ if [ "${behind}" -eq 0 ]; then
+ exit 0
+ fi
+ echo "::notice title=This run's merge ref is ${behind} commit(s) behind ${BASE_REF}::A pull_request run tests a merge of this branch into ${BASE_REF} as it stood when the run was created, and ${BASE_REF} has gained ${behind} commit(s) since. If this failure was fixed on ${BASE_REF} in that window, the fix is not in this run, and re-running will not pick it up because a re-run replays the same merge commit. Push to the branch to have the merge ref rebuilt. This notice does not mean the failure is spurious."
diff --git a/.github/workflows/studio-backend-ci.yml b/.github/workflows/studio-backend-ci.yml
index 2321d6d36e0..5eacca9d927 100644
--- a/.github/workflows/studio-backend-ci.yml
+++ b/.github/workflows/studio-backend-ci.yml
@@ -437,6 +437,14 @@ jobs:
tests/test_recommended_folders_permission.py \
tests/test_hf_cache_settings.py
+ - name: Why a failure here might already be fixed on the base branch
+ # `if: failure()` so a green job never runs it, and the action itself stays
+ # quiet unless the base branch has actually moved since this run's merge ref
+ # was built. See .github/actions/merge-ref-age for the three pull requests
+ # that were misread this way.
+ if: failure() && github.event_name == 'pull_request'
+ uses: ./.github/actions/merge-ref-age
+
- name: Save the pip cache
if: always()
uses: ./.github/actions/pip-cache-save
@@ -719,6 +727,14 @@ jobs:
if: matrix.shard == 'rest'
run: python tests/validate_studio_features.py
+ - name: Why a failure here might already be fixed on the base branch
+ # `if: failure()` so a green job never runs it, and the action itself stays
+ # quiet unless the base branch has actually moved since this run's merge ref
+ # was built. See .github/actions/merge-ref-age for the three pull requests
+ # that were misread this way.
+ if: failure() && github.event_name == 'pull_request'
+ uses: ./.github/actions/merge-ref-age
+
- name: Save the pip cache
if: always()
uses: ./.github/actions/pip-cache-save
diff --git a/.github/workflows/workflow-trigger-lint.yml b/.github/workflows/workflow-trigger-lint.yml
index fa466866bc7..6f7583c9c6b 100644
--- a/.github/workflows/workflow-trigger-lint.yml
+++ b/.github/workflows/workflow-trigger-lint.yml
@@ -160,6 +160,7 @@ jobs:
tests/studio/test_main_runs_survive_merge_bursts.py \
tests/studio/test_pull_requests_cancel_superseded_runs.py \
tests/studio/test_probe_cost_claim_is_measured.py \
+ tests/studio/test_merge_ref_age_is_reported.py \
tests/studio/test_pester_bootstrap_hardening.py \
tests/studio/test_playwright_suites_run_in_ci.py \
tests/studio/test_sdk_installs_are_major_bounded.py \
diff --git a/tests/studio/test_merge_ref_age_is_reported.py b/tests/studio/test_merge_ref_age_is_reported.py
new file mode 100644
index 00000000000..efd4d0074cf
--- /dev/null
+++ b/tests/studio/test_merge_ref_age_is_reported.py
@@ -0,0 +1,104 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
+
+"""A failing pytest job says how old its merge ref is, and saying so cannot fail the job.
+
+A `pull_request` run tests `refs/pull/N/merge`, a merge of the branch into the base as
+the base stood when the run was created. GitHub does not rebuild that ref because the
+base later moved, and a re-run replays the same merge commit, so a fix that lands on
+main after a run starts is invisible to that run for as long as the branch sits still.
+
+That cost real time: #11372 added `encoding = "utf-8"` to three test files and merged at
+14:35 UTC, and runs created at 13:39, 14:26 and 14:29 on #11375, #11357 and #11373 went
+on failing `tests/test_source_read_encoding.py` on exactly those three lines. All three
+branches had never touched the files.
+
+`.github/actions/merge-ref-age` prints a notice that says so. What this file keeps is
+the two properties that make it worth having, because both are easy to lose:
+
+ 1. Every job here that runs pytest carries it, and carries it gated on `failure()`.
+ A job added later that does not is a job where the same confusion returns.
+
+ 2. It cannot fail a job. It runs exactly when a job is already failing, so a bug in
+ it would land as a second, misleading error on top of the real one. The action's
+ script therefore never exits non-zero and never emits `::error::`.
+
+It deliberately does NOT check the wording of the notice. The message is prose, prose
+guards have to parse English to be right, and the value here is that the step exists and
+is harmless, not that any particular sentence survives.
+"""
+
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+import pytest
+import yaml
+
+ROOT = Path(__file__).resolve().parents[2]
+WORKFLOW = ROOT / ".github" / "workflows" / "studio-backend-ci.yml"
+ACTION = ROOT / ".github" / "actions" / "merge-ref-age" / "action.yml"
+
+_USES = "./.github/actions/merge-ref-age"
+
+
+def _jobs() -> dict:
+ return yaml.safe_load(WORKFLOW.read_text(encoding = "utf-8"))["jobs"]
+
+
+def _jobs_that_run_pytest() -> list[str]:
+ names = []
+ for name, job in _jobs().items():
+ if any("pytest" in str(step.get("run", "")) for step in job.get("steps", [])):
+ names.append(name)
+ return names
+
+
+def test_some_job_here_runs_pytest():
+ """Rule 1 below is vacuous if nothing matches it, which is how such rules die."""
+ assert _jobs_that_run_pytest(), (
+ f"{WORKFLOW.name} no longer has a job that runs pytest, so either the suites "
+ f"moved and this guard should move with them, or it is reading the wrong file"
+ )
+
+
[email protected]("job_name", _jobs_that_run_pytest())
+def test_every_pytest_job_reports_the_age_of_its_merge_ref(job_name):
+ steps = _jobs()[job_name]["steps"]
+ reporting = [step for step in steps if step.get("uses") == _USES]
+ assert reporting, (
+ f"job {job_name!r} in {WORKFLOW.name} runs pytest but does not use {_USES}. A "
+ f"failure in it cannot tell the reader that the base branch moved since the "
+ f"merge ref was built, which is how a fix that already landed gets debugged "
+ f"a second time"
+ )
+ for step in reporting:
+ condition = str(step.get("if", ""))
+ assert "failure()" in condition, (
+ f"job {job_name!r} runs {_USES} under {condition!r}. It is meant to speak "
+ f"only when the job has already failed; on a green job the notice is noise "
+ f"and the API call is waste"
+ )
+
+
+def test_reporting_the_age_cannot_fail_the_job():
+ """It runs on the failure path, where a second error would bury the first one."""
+ action = ACTION.read_text(encoding = "utf-8")
+ body = yaml.safe_load(action)
+ scripts = [str(step.get("run", "")) for step in body["runs"]["steps"]]
+ assert any(scripts), f"{ACTION} has no script left to check"
+
+ for script in scripts:
+ # Comments explain the exits; the code is what runs.
+ code = "\n".join(line for line in script.splitlines() if not line.lstrip().startswith("#"))
+ bad = re.findall(r"^\s*exit\s+(?!0\b)\S+", code, re.M)
+ assert not bad, (
+ f"{ACTION} exits non-zero ({bad}). It runs only when the job is already "
+ f"failing, so a non-zero exit here adds a second error on top of the real "
+ f"one and points the reader at the wrong thing"
+ )
+ assert "::error" not in code, (
+ f"{ACTION} emits ::error::. Advisory only: a stale merge ref does not make "
+ f"the failure untrue, and annotating it as an error says it does"
+ )