Skip to content

Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.

Kaggle T4: Latest_compile and Vision_FLA_compile legs, plus gpt-oss completions and GGUF (#9700)

Daniel Han40f452effaae
diff --git a/.github/scripts/kaggle_prefetch.py b/.github/scripts/kaggle_prefetch.py
new file mode 100644
index 00000000000..958a4554162
--- /dev/null
+++ b/.github/scripts/kaggle_prefetch.py
@@ -0,0 +1,260 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""The model prefetch that both Kaggle kernels paste into a generated cell.
+
+A Kaggle GPU session is 2xT4 and the kernel keeps both busy training, but
+downloading a model is CPU and network work that holds a card idle for its
+whole duration. This module is the body of a lane that does that downloading
+EARLY, beside the training legs, so the leg that needs the model finds it
+already on disk.
+
+Why a module and not a copy in each builder: `kaggle_t4_ci` prefetches the leg
+models into the Kaggle image's default cache, and `kaggle_studio_ci` prefetches
+Studio's two models into Studio's own private ``HF_HOME``. Same retry policy,
+same reporting, two different cache roots. One copy that takes the root as an
+argument is the only version of that which stays in agreement with itself.
+
+Load it BY PATH (``importlib.util.spec_from_file_location``), never with a
+plain ``import``. Both script directories already ship a ``build_kernel.py``
+and a ``report.py``, the test suite puts both on ``sys.path``, and a plain
+import therefore resolves to whichever reached ``sys.modules`` first -- which
+is decided by test order rather than by intent. That collision has been paid
+for here once already: one ``sys.path.insert`` added for a single test took
+nine unrelated tests down with it.
+
+WHAT THIS IS NOT: it is not a correctness mechanism. Every caller must treat a
+failed prefetch as a no-op, because the payload that wants the model downloads
+it for itself exactly as it did before this existed. A prefetch that fails the
+kernel would be a new way to go red for something that is not under test.
+"""
+
+from __future__ import annotations
+
+# The sentinel the driver and the reporters grep for. One record per repo, on
+# its own line, so `kernel.log` ALONE measures the download -- which is the
+# number the whole schedule is built around and the one thing no artifact has
+# ever separated from weight-load time.
+PREFETCH_SENTINEL = "KAGGLE_CI_PREFETCH"
+
+
+def _normalise(repos):
+ """``["a", ("b", ["*.gguf"])]`` -> ``[("a", None), ("b", ["*.gguf"])]``.
+
+ A bare string means the WHOLE repo, which is right for a small model whose
+ every file gets loaded and wrong for anything with variants. Run
+ 32667451396 fetched 69.1 GB of ``Qwen3.5-2B-GGUF`` -- every quant in the
+ repo -- so that Studio could load one UD-Q4_K_XL file, and 55.1 GB of a
+ checkpoint that was never opened at all. On a 4-core Kaggle box that is not
+ just wasted bandwidth: it is CPU stolen from the payloads the prefetch
+ exists to speed up, and it pushed the Studio install from 258s to 673.5s.
+ """
+ out = []
+ for entry in repos:
+ if isinstance(entry, str):
+ out.append((entry, None))
+ continue
+ repo, patterns = entry
+ out.append((repo, list(patterns) if patterns else None))
+ return out
+
+
+def prefetch_cell(
+ repos: list,
+ *,
+ hf_home: str | None = None,
+ attempt_timeout: int = 900,
+ total_timeout: int = 1800,
+) -> str:
+ """Source for a cell (or a driver thread) that warms ``repos``, in order.
+
+ ``repos`` is ordered and the order is load bearing: the caller puts the
+ repo with the longest lead time first, because a prefetch only pays for
+ the work it finishes BEFORE the payload that wants it starts.
+
+ ``hf_home`` of None means "do not touch HF_HOME", which is what the leg
+ prefetch needs -- the legs read the Kaggle image's default cache and the
+ entire point is to land in the cache they read. Setting it to a private
+ directory there would produce a perfectly healthy prefetch that no payload
+ can see, a full 12 GB of work thrown away, and a green run.
+ """
+ # repr(), NOT json.dumps(). This text is Python, and `json.dumps(None)` is
+ # `null`, which parses fine and dies with a NameError the first time the
+ # cell RUNS -- on a Kaggle session, minutes in, having already paid for the
+ # box. `test_the_generated_prefetch_cell_runs` exists because compiling the
+ # cell did not catch exactly that.
+ return f'''
+import json, os, threading, time
+
+_REPOS = {_normalise(repos)!r}
+_HF_HOME = {hf_home!r}
+_ATTEMPT_TIMEOUT = {attempt_timeout}
+_TOTAL_TIMEOUT = {total_timeout}
+_DEADLINE = time.time() + _TOTAL_TIMEOUT
+
+if _HF_HOME:
+ os.environ["HF_HOME"] = _HF_HOME
+
+
+def _repo_bytes(repo):
+ """Size of THIS repo's directory in the hub cache.
+
+ Measured rather than taken from the return value, because
+ `snapshot_download` reports a path and not a transfer size, and a repo that
+ was ALREADY warm has to read as zero new bytes rather than as its own size
+ or every rerun looks like a full download.
+
+ Scoped to the one repo rather than differencing the whole cache, for two
+ reasons. The cheap one: walking a cache holding a 12 GB model, twice per
+ repo, is real time charged to the very measurement it is taking. The one
+ that matters: the legs are downloading into this same cache CONCURRENTLY,
+ so a whole-cache delta silently credits their bytes to this lane and
+ reports a download rate the Hub never delivered.
+
+ Counted ONCE per blob. A hub snapshot is a tree of symlinks into `blobs/`,
+ so `os.stat` follows the link and the same bytes are then counted again
+ under the blob's own name -- roughly 2x, which is not a rounding error in a
+ number whose whole purpose is a MB/s rate. `lstat` measures the link
+ itself, and the inode set also collapses hardlinks, which is what a
+ filesystem without symlink support gives instead.
+ """
+ root = os.environ.get("HF_HOME") or os.path.expanduser("~/.cache/huggingface")
+ folder = "models--" + repo.replace("/", "--")
+ total = 0
+ seen = set()
+ for dirpath, _dirnames, filenames in os.walk(
+ os.path.join(root, "hub", folder), followlinks = False
+ ):
+ for name in filenames:
+ path = os.path.join(dirpath, name)
+ if os.path.islink(path):
+ continue
+ try:
+ info = os.lstat(path)
+ except OSError:
+ continue
+ key = (info.st_dev, info.st_ino)
+ if key in seen:
+ continue
+ seen.add(key)
+ total += info.st_size
+ return total
+
+
+def _attempt(repo, patterns, disable_xet):
+ """One `snapshot_download`, in a thread, under a wall-clock watchdog.
+
+ The watchdog is the point, and it is not the same thing as a retry. Xet
+ classifies 408/429/5xx as transient and retries them itself with backoff
+ (5 attempts, 3s base, a six-minute cap per delay), so a throttled or
+ stalling transfer can sit inside ONE call for many minutes without ever
+ raising anything for an ordinary `except` to catch. There are documented
+ cases of Xet stalling where classic HTTP ran at line speed. So an attempt
+ that stops making progress has to be abandoned on the clock and retried on
+ a DIFFERENT transport; retrying the same stalling transport is how a retry
+ loop turns into a way to spend the whole session.
+
+ The thread is left running (daemon) rather than killed -- Python cannot
+ kill it -- and its bytes are not lost: since huggingface_hub 1.18.0 each
+ file lands through a process-unique temporary and an atomic move, and the
+ files it already completed stay valid in the cache for the next attempt.
+ """
+ env_backup = os.environ.get("HF_HUB_DISABLE_XET")
+ if disable_xet:
+ os.environ["HF_HUB_DISABLE_XET"] = "1"
+ box = {{}}
+
+ def _run():
+ try:
+ from huggingface_hub import snapshot_download
+ snapshot_download(repo_id=repo, allow_patterns=patterns)
+ box["ok"] = True
+ except BaseException as exc: # noqa: BLE001
+ box["error"] = f"{{type(exc).__name__}}: {{exc}}"
+
+ t = threading.Thread(target=_run, daemon=True)
+ t.start()
+ budget = min(_ATTEMPT_TIMEOUT, max(1.0, _DEADLINE - time.time()))
+ t.join(budget)
+ if env_backup is None:
+ os.environ.pop("HF_HUB_DISABLE_XET", None)
+ else:
+ os.environ["HF_HUB_DISABLE_XET"] = env_backup
+ if t.is_alive():
+ return False, f"no result within {{budget:.0f}}s"
+ if box.get("ok"):
+ return True, None
+ return False, box.get("error", "unknown")
+
+
+def prefetch_all():
+ for repo, patterns in _REPOS:
+ started = time.time()
+ before = _repo_bytes(repo)
+ ok, error, transport, attempts = False, None, None, 0
+ # Time inside the attempt that WORKED, separately from the wall clock
+ # of the whole repo. They differ by the backoff sleeps and the failed
+ # attempts, and conflating them corrupts the one number this lane
+ # exists to produce: a run that retried twice would report its 12 GB as
+ # having taken 9s longer than it did and understate the achieved rate
+ # accordingly. `seconds` is what the SCHEDULE waits for; `download_
+ # seconds` is how fast the Hub actually was.
+ download_seconds = None
+ # Last attempt forces classic HTTP. The first two keep whatever
+ # transport the hub chose (Xet when the repo has Xet metadata and
+ # hf_xet is installed, which since huggingface_hub 0.32 is the default
+ # and is therefore live here whether or not anyone chose it).
+ _plan = (False, False, True)
+ for _i, disable_xet in enumerate(_plan):
+ if time.time() >= _DEADLINE:
+ error = "the prefetch budget was spent before this repo"
+ break
+ attempts += 1
+ _t0 = time.time()
+ ok, error = _attempt(repo, patterns, disable_xet)
+ transport = "http" if disable_xet else "auto"
+ if ok:
+ download_seconds = round(time.time() - _t0, 1)
+ break
+ print(f"{PREFETCH_SENTINEL}_RETRY " + json.dumps(
+ {{"repo": repo, "attempt": attempts, "error": str(error)[:300]}}),
+ flush=True)
+ # Backoff only when something will actually follow it. Sleeping
+ # after the LAST attempt buys nothing -- there is no retry left to
+ # space out -- and it is the difference between this lane giving up
+ # in 9s and giving up in 18s, per repo, on the path where the
+ # network is already known to be unhappy.
+ #
+ # Bounded by what is LEFT of the budget too, not just by the curve:
+ # a lane already at its deadline that still sleeps 15s is spending
+ # session time to accomplish nothing.
+ if _i + 1 < len(_plan):
+ time.sleep(max(0.0, min(15.0, 3.0 * attempts, _DEADLINE - time.time())))
+ seconds = round(time.time() - started, 1)
+ moved = max(0, _repo_bytes(repo) - before)
+ print(f"{PREFETCH_SENTINEL} " + json.dumps({{
+ "repo": repo, "ok": bool(ok), "seconds": seconds,
+ "download_seconds": download_seconds, "bytes": moved,
+ "mb_per_s": (round(moved / 1e6 / download_seconds, 1)
+ if download_seconds else None),
+ # Reported so an over-narrow filter is visible rather than silent.
+ # A pattern that matches nothing downloads nothing, reports ok and
+ # leaves the payload to fetch the model itself -- a prefetch that
+ # looks perfect and does nothing. `bytes` next to `patterns` is
+ # what makes that readable in the summary.
+ "patterns": patterns,
+ "transport": transport, "attempts": attempts,
+ "error": None if ok else str(error)[:300],
+ }}), flush=True)
+
+
+# Never raises. A prefetch is an optimisation: the payload that wants the model
+# still downloads it for itself, so a failure here costs seconds, and letting
+# it propagate would invent a way for the kernel to go red for something that
+# is not under test.
+try:
+ prefetch_all()
+except BaseException as exc: # noqa: BLE001
+ print(f"{PREFETCH_SENTINEL}_ABORTED " + json.dumps(
+ {{"error": f"{{type(exc).__name__}}: {{exc}}"}}), flush=True)
+'''
diff --git a/.github/scripts/kaggle_studio_ci/build_kernel.py b/.github/scripts/kaggle_studio_ci/build_kernel.py
index f5b187d3322..6ef1a51ed15 100644
--- a/.github/scripts/kaggle_studio_ci/build_kernel.py
+++ b/.github/scripts/kaggle_studio_ci/build_kernel.py
@@ -115,8 +115,106 @@ def _code_cell(source: str) -> dict:
}
-def build_payload_notebook(*, unsloth_ref: str, repo_url: str, payload_args: str) -> dict:
- """The notebook that installs Unsloth and runs the payload against it."""
+def _prefetch_builder():
+ """Load ``kaggle_prefetch.py`` by PATH. See the note in
+ ``kaggle_t4_ci/build_kernel.py``: two sibling script directories both ship
+ a ``build_kernel`` and a ``report``, so a plain import here resolves by
+ whichever landed in ``sys.modules`` first.
+ """
+ import importlib.util
+
+ path = Path(__file__).resolve().parents[1] / "kaggle_prefetch.py"
+ spec = importlib.util.spec_from_file_location("kaggle_ci__prefetch", path)
+ if spec is None or spec.loader is None:
+ raise ImportError(f"cannot load the prefetch builder from {path}")
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+def _models_from(payload_args: str) -> list[str]:
+ """The repos this payload will load, read off its own argv.
+
+ NOT a second copy of the defaults. ``--chat-model`` and ``--train-model``
+ are dispatch inputs, so a hardcoded pair here would prefetch the wrong
+ models the moment anyone used them -- and prefetching the wrong repo is
+ invisible: it downloads happily, warms a cache nobody reads, and reports
+ success.
+ """
+ tokens = payload_args.split()
+ picked = {}
+ for flag, default in (
+ # These defaults must track run_studio_gpu.py's own, or the prefetch
+ # warms a cache the payload never reads -- which downloads happily and
+ # reports success. tests/kaggle/test_t4_ci_transport.py compares the
+ # two, which is how this pair was caught drifting.
+ ("--chat-model", "unsloth/Qwen3.5-2B-MTP-GGUF"),
+ ("--train-model", "unsloth/Qwen3.5-2B"),
+ # Read for the same reason as the repos: Studio loads ONE quant out of
+ # a GGUF repo that ships many, and an unfiltered snapshot pulls all of
+ # them. Run 32667451396 fetched 69.1 GB of Qwen3.5-2B-GGUF to serve a
+ # single UD-Q4_K_XL file. Taken off argv rather than hardcoded so a
+ # dispatch that overrides the variant filters on the variant it chose.
+ ("--chat-variant", "UD-Q4_K_XL"),
+ ):
+ value = default
+ for i, token in enumerate(tokens):
+ if token == flag and i + 1 < len(tokens):
+ value = tokens[i + 1]
+ elif token.startswith(flag + "="):
+ value = token.split("=", 1)[1]
+ picked[flag] = value
+ # Chat model first: it is the GGUF that llama.cpp has to serve, and it is
+ # the larger of the two.
+ #
+ # The variant glob is deliberately loose at both ends. Multi-part GGUFs are
+ # named `...UD-Q4_K_XL-00001-of-00002.gguf`, so anchoring the suffix would
+ # match the single-file case and silently miss every shard of the split
+ # one -- which downloads nothing, reports success, and leaves Studio to
+ # fetch it itself.
+ variant = picked["--chat-variant"]
+ chat = (picked["--chat-model"], [f"*{variant}*"]) if variant else picked["--chat-model"]
+ return [chat, picked["--train-model"]]
+
+
+def build_payload_notebook(
+ *,
+ unsloth_ref: str,
+ repo_url: str,
+ payload_args: str,
+ phase: str | None = None,
+) -> dict:
+ """The notebook that installs Unsloth and runs the payload against it.
+
+ ``phase`` splits that notebook in two, for the merged kernel that runs this
+ payload beside the T4 notebook legs (see ``kaggle_t4_ci/build_kernel.py``).
+ The split point is not arbitrary: everything up to and including the
+ Playwright install is checkout, download and compile, none of which touches
+ a GPU, and the ``verify`` cell is the first thing that requires one -- it
+ refuses with "no CUDA device in the Studio venv" when
+ ``torch.cuda.is_available()`` is False. So
+
+ * ``"install"`` is the GPU-free prefix and can run while both cards are
+ busy training,
+ * ``"test"`` is everything that needs a card, and runs once they are free.
+
+ ``None`` builds the whole thing as one notebook, which is what the
+ standalone Studio workflow still does.
+
+ The two halves communicate through the DISK, not through the interpreter:
+ ``setup`` recomputes the same paths in both (``_pick_work_root`` is
+ deterministic within a session) and the test half re-derives ``VENV_PY``
+ from ``STUDIO_HOME`` rather than inheriting it.
+
+ One trap that is easy to walk into here: the install half must still SEE
+ both GPUs. ``install.sh --local`` resolves torch, and a CPU-only torch
+ resolved by an installer that could not find a device is precisely the
+ regression the verify cell exists to catch. So the caller leaves
+ ``CUDA_VISIBLE_DEVICES`` unset on that lane rather than blanking it; the
+ install reads device capability and never allocates.
+ """
+ if phase not in (None, "install", "test"):
+ raise ValueError(f"phase must be None, 'install' or 'test', not {phase!r}")
setup = f"""# Where everything lives.
#
@@ -338,15 +436,60 @@ print("{PAYLOAD_SENTINEL} complete rc=" + str(proc.returncode), flush=True)
# aborting here would lose the cells below it.
"""
+ # Studio's two models, fetched on the half that is ALREADY hidden.
+ #
+ # Both were previously pulled inside run_studio_gpu.py, which is the TEST
+ # half, so the merged kernel hid Studio's clone, pip and Playwright browser
+ # and then paid full price for its downloads with both cards idle. They go
+ # here instead, under Studio's own HF_HOME -- which is why this cannot use
+ # the t4 driver's lane: that one deliberately targets the image default so
+ # the training legs can read it, and Studio's install is a user-shaped
+ # install with a cache root of its own.
+ #
+ # Last in the install phase, after the venv and the browser: those are what
+ # the test half cannot start without, and a download that overruns the card
+ # queue must not be what delays them.
+ #
+ # hf_home=None means "inherit", NOT "use the default". The setup cell runs
+ # first in this same notebook and has already put Studio's private root in
+ # os.environ["HF_HOME"], so inheriting is how this lands there. Passing the
+ # path again would be a second copy of _pick_work_root's answer, free to
+ # disagree with the real one. `test_the_studio_prefetch_lands_in_studios
+ # _own_cache` pins the ordering that makes inheriting correct.
+ prefetch = _prefetch_builder().prefetch_cell(
+ _models_from(payload_args),
+ hf_home = None,
+ attempt_timeout = 600,
+ total_timeout = 1200,
+ )
+
+ # Marks the GPU-free half done, on its own line, so the driver can gate the
+ # test half on a sentinel it saw rather than on a returncode alone.
+ installed = f"""print("{PAYLOAD_SENTINEL} INSTALLED " + json.dumps({{
+ "studio_home": str(STUDIO_HOME), "venv": str(VENV_PY),
+}}), flush=True)
+"""
+
+ # Re-derives what the install half left on disk. VENV_PY is defined in the
+ # install cell, which the test half does not carry, so without this the
+ # verify cell below dies on a NameError rather than on anything it tests.
+ bridge = f"""VENV_PY = STUDIO_HOME / "unsloth_studio" / "bin" / "python"
+if not VENV_PY.is_file():
+ fail_report(f"the install phase left no interpreter at {{VENV_PY}}; it either "
+ f"did not run or did not land in the directory this half looks in")
+ raise SystemExit(f"no interpreter at {{VENV_PY}}")
+print("{PAYLOAD_SENTINEL} venv " + str(VENV_PY), flush=True)
+"""
+
+ if phase == "install":
+ cells = [setup, clone, install, browser, prefetch, installed]
+ elif phase == "test":
+ cells = [setup, bridge, verify, run]
+ else:
+ cells = [setup, clone, install, browser, prefetch, verify, run]
+
return {
- "cells": [
- _code_cell(setup),
- _code_cell(clone),
- _code_cell(install),
- _code_cell(browser),
- _code_cell(verify),
- _code_cell(run),
- ],
+ "cells": [_code_cell(source) for source in cells],
"metadata": {
"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},
"language_info": {"name": "python"},
diff --git a/.github/scripts/kaggle_studio_ci/report.py b/.github/scripts/kaggle_studio_ci/report.py
index b64eebdfaf3..916fb97b7c9 100644
--- a/.github/scripts/kaggle_studio_ci/report.py
+++ b/.github/scripts/kaggle_studio_ci/report.py
@@ -141,9 +141,46 @@ def render(report: dict) -> list[str]:
return lines
+# The label this payload reports under. Duplicated in kaggle_t4_ci/report.py;
+# see the note there for why it is not shared.
+STUDIO_LABEL = "studio-gpu"
+
+
+def own_verdict(kernel_verdict: str, kernel_reason: str, reports: list, expect: int):
+ """This reporter's verdict over ITS OWN payloads, not the kernel's.
+
+ The launcher writes one verdict for the whole kernel, and since
+ --with-studio that kernel holds two unrelated experiments. Reading the
+ kernel verdict here means a failing training leg prints "Studio GPU smoke: FAIL"
+ above a section listing zero failures, and a failing Studio payload prints
+ the same over four green legs. Both are the misleading-red twin of the
+ green tick that tested nothing, and both would send someone to read the
+ wrong payload.
+
+ So the verdict is recomputed from the filtered reports. The kernel reason
+ is kept only when the two agree; otherwise it describes the other half.
+
+ `infra` is deliberately not synthesised: with nothing of ours back, the
+ kernel-level reason (quota, concurrency cap, a push that was throttled) is
+ the only account of why, and it applies to every payload equally.
+ """
+ if not reports:
+ return (kernel_verdict if kernel_verdict == "infra" else "partial"), kernel_reason
+ failing = [r for r in reports if not r.get("passed")]
+ if failing:
+ return "fail", f"{len(failing)} of {len(reports)} payload(s) failed their assertions"
+ if len(reports) < expect:
+ return "partial", f"only {len(reports)} of {expect} payload(s) reported back"
+ return "pass", f"all {len(reports)} payload(s) passed"
+
+
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--evidence", required = True)
+ # One, always: the payload's two notebooks are halves of one experiment and
+ # produce exactly one `studio-gpu` report between them. See the count note
+ # in kaggle_t4_ci/build_kernel.py's --all-kernels tail.
+ ap.add_argument("--expect", type = int, default = 1)
args = ap.parse_args()
evidence = Path(args.evidence)
@@ -162,6 +199,15 @@ def main() -> int:
reason = result.get("reason", "")
reports = result.get("reports", [])
+ # This payload can share a kernel with the T4 notebook legs (see
+ # kaggle_t4_ci/build_kernel.py --with-studio), and every payload in that
+ # kernel reports through the same prefix. The legs are a different shape --
+ # a per-step metric trace rather than assertions -- so rendering them here
+ # would produce Studio sections describing training runs. Each reporter
+ # owns its own labels.
+ reports = [r for r in reports if r.get("label") == STUDIO_LABEL]
+ verdict, reason = own_verdict(verdict, reason, reports, args.expect)
+
header = {
"pass": "### Unsloth GPU smoke: PASS",
"fail": "### Unsloth GPU smoke: FAIL",
diff --git a/.github/scripts/kaggle_t4_ci/build_kernel.py b/.github/scripts/kaggle_t4_ci/build_kernel.py
index 9bc646266e9..c17ee5daff3 100644
--- a/.github/scripts/kaggle_t4_ci/build_kernel.py
+++ b/.github/scripts/kaggle_t4_ci/build_kernel.py
@@ -56,7 +56,14 @@ from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
-from legs import KERNELS, PACKAGE_UNDER_TEST, Leg, expand_install, resolve # noqa: E402
+from legs import ( # noqa: E402
+ KERNELS,
+ PACKAGE_UNDER_TEST,
+ PREFETCH_REPOS,
+ Leg,
+ expand_install,
+ resolve,
+)
DRIVER_SENTINEL = "KAGGLE_T4_CI_DRIVER"
PAYLOAD_SENTINEL = "KAGGLE_T4_CI_PAYLOAD"
@@ -145,8 +152,16 @@ def build_payload_notebook(
extra_args: tuple[str, ...] = (),
reference: str | None = None,
) -> dict:
- """A single-GPU notebook that installs one leg's stack and runs it."""
+ """A notebook that installs one leg's stack and runs it.
+
+ One GPU for every leg but the `all_cards` one, which is unpinned so that
+ unsloth's DEVICE_COUNT > 1 code path is reachable at all.
+ """
root = _kernel_root(leg)
+ # Baked in HERE, at build time, from the leg's own declaration. The check
+ # in the verify cell compares against this constant rather than against
+ # anything the running kernel can see.
+ expected_visible = 2 if leg.all_cards else 1
wanted = list(leg.files)
if leg.entry not in wanted:
wanted.append(leg.entry)
@@ -185,12 +200,45 @@ print("{PAYLOAD_SENTINEL} sources " + json.dumps(sorted(FILES)), flush=True)
# differs between the control leg and the version canary. They are printed
# before they run so the kernel log alone says what was asked for, which is
# what makes a canary failure attributable without downloading anything.
-import json, subprocess, sys, time
+import glob, json, os, re, subprocess, sys, time
print("{PAYLOAD_SENTINEL} leg {leg.name}: {leg.summary}", flush=True)
GROUPS = {json.dumps(groups)}
+UNINSTALL = {json.dumps(list(leg.uninstall))}
print("{PAYLOAD_SENTINEL} install plan " + json.dumps(GROUPS), flush=True)
+# Wheels the driver built ONCE, before any leg started, from the very SHAs
+# these groups name. Empty or missing means the build did not happen or did not
+# finish, and every spec below then resolves exactly as it did before -- a
+# failed wheel build costs its own time and changes nothing about what is
+# tested.
+WHEEL_DIR = os.environ.get("UNSLOTH_CI_WHEEL_DIR", "")
+
+
+# `name @ git+...@sha` -> the prebuilt wheel for `name`, when there is one.
+#
+# Matched on the DISTRIBUTION NAME, normalised the way a wheel filename is
+# (PEP 503: runs of -_. collapse to _), because `unsloth_zoo` on the left of the
+# `@` arrives as `unsloth_zoo-2026.8.1-py3-none-any.whl` on disk. Getting that
+# wrong is invisible: no wheel matches, every leg falls back to its git spec,
+# and the run is merely as slow as it was before.
+#
+# A comment and not a docstring: this whole cell is generated from an f-string
+# delimited by triple quotes, so a docstring here ENDS the template and the
+# build dies with a SyntaxError in build_kernel.py rather than in the cell.
+def _localise(spec):
+ if not WHEEL_DIR or "@ git+" not in spec:
+ return spec
+ name = spec.split("@ git+", 1)[0].strip()
+ key = re.sub(r"[-_.]+", "_", name).lower()
+ for cand in sorted(glob.glob(os.path.join(WHEEL_DIR, "*.whl"))):
+ stem = os.path.basename(cand).split("-", 1)[0]
+ if re.sub(r"[-_.]+", "_", stem).lower() == key:
+ return cand
+ return spec
+
+
def pip(args):
+ args = [_localise(a) for a in args]
cmd = [sys.executable, "-m", "pip", "install", "-q", *args]
print(" $ " + " ".join(cmd[3:]), flush=True)
# github.com occasionally 500s on a git fetch, and PyPI occasionally
@@ -228,7 +276,35 @@ def pip(args):
raise SystemExit(f"pip install failed: {{args}}")
for group in GROUPS:
+ _g0 = time.time()
pip(group)
+ # Per group, because "install took 191s" was all any artifact ever said and
+ # it is not actionable: it hides which of the four groups was the cost, and
+ # therefore whether the shared wheels helped at all.
+ print("{PAYLOAD_SENTINEL} install group " + json.dumps({{
+ "seconds": round(time.time() - _g0, 1),
+ "spec": [_localise(a) for a in group],
+ }}), flush=True)
+# After the install groups, never before: these distributions arrive as
+# dependencies of something installed above, so removing them earlier would
+# remove nothing and the later install would put them back.
+#
+# Never fatal. If a name is absent pip says so and exits 0, and even a genuine
+# failure here should not cost the session -- the payload then fails on its own
+# terms with an error that names what actually went wrong, which is more useful
+# than a stand-down during setup.
+if UNINSTALL:
+ _u0 = time.time()
+ _up = subprocess.run(
+ [sys.executable, "-m", "pip", "uninstall", "-y", *UNINSTALL],
+ capture_output=True, text=True)
+ print("{PAYLOAD_SENTINEL} uninstall " + json.dumps({{
+ "seconds": round(time.time() - _u0, 1),
+ "spec": UNINSTALL,
+ "returncode": _up.returncode,
+ "stdout": _up.stdout[-2000:],
+ "stderr": _up.stderr[-2000:],
+ }}), flush=True)
print("{PAYLOAD_SENTINEL} install done", flush=True)
"""
@@ -325,12 +401,23 @@ try:
"name": torch.cuda.get_device_name(0) if torch.cuda.is_available() else None,
"visible": __import__("os").environ.get("CUDA_VISIBLE_DEVICES"),
}}), flush=True)
- # Exactly one GPU must be visible: two would mean the driver failed to pin
- # this payload to its own card, and accelerate would shard across both,
- # which is a different test from the single T4 a Colab user gets. Zero
- # means the payload cannot run at all on the hardware it was billed for.
- assert torch.cuda.device_count() == 1, (
- f"expected exactly 1 visible GPU, got {{torch.cuda.device_count()}}")
+ # EXACTLY the count this leg asked for, and the number is baked in at build
+ # time rather than read from the environment -- reading it from the same
+ # variable that sets it would make the check agree with whatever happened.
+ #
+ # 1 for every ordinary leg: two would mean the driver failed to pin this
+ # payload to its own card, and accelerate would shard across both, which is
+ # a different test from the single T4 a Colab user gets. Zero means the
+ # payload cannot run at all on the hardware it was billed for.
+ #
+ # 2 for the `multi_gpu` leg, which is deliberately unpinned so that
+ # unsloth's DEVICE_COUNT > 1 branch is reachable. A leg that asks for two
+ # and gets one still fails here, which is the case that matters: silently
+ # running it on one card would leave the multi-GPU claim asserted and
+ # untested, and green.
+ assert torch.cuda.device_count() == {expected_visible}, (
+ f"expected exactly {expected_visible} visible GPU(s), got "
+ f"{{torch.cuda.device_count()}}")
except BaseException as exc:
detail = f"{{type(exc).__name__}}: {{exc}}"
print("{PAYLOAD_SENTINEL} GPU_UNUSABLE " + json.dumps(detail), flush=True)
@@ -494,6 +581,14 @@ def build_driver(
per_run_timeout: int,
isolation: dict[str, bool] | None = None,
expected_gpus: int = SESSION_GPUS,
+ cpu_lane: str | None = None,
+ after_gpu: str | None = None,
+ prefetch_repos: tuple[str, ...] = (),
+ vram_source: dict | None = None,
+ after_gpu_concurrent: bool = False,
+ shared_wheel_specs: tuple[str, ...] = (),
+ overlays: dict[str, tuple[str, ...]] | None = None,
+ all_card: tuple[str, ...] = (),
) -> dict:
"""Kernel notebook that runs the payloads across the session's GPUs.
@@ -504,23 +599,84 @@ def build_driver(
``payloads`` may hold MORE entries than there are cards. They queue: one
worker per card takes the next leg only when its current one has exited, so
a card carries exactly one payload at a time. ``payloads`` is ordered, and
- that order is the start order -- longest leg first, since the longest leg
- is what sets the makespan and a greedy scheduler cannot balance around one
- it picks up last.
+ that order is the start order. It comes from ``legs.KERNELS``, which
+ documents why gptoss sits third rather than first; do not sort it here.
+
+ ``prefetch_repos`` are warmed, in the order given, on a lane that takes no
+ card and no virtualenv. It runs in the driver's own interpreter using the
+ Kaggle image's ``huggingface_hub``, so it starts at t=0 rather than after a
+ venv build, and it deliberately does NOT set ``HF_HOME``: the legs read the
+ image default and the whole point is to land in the cache they read.
``expected_gpus`` is how many cards the packing was built for. It is what
the kernel stands down against, and it is deliberately NOT ``len(payloads)``
any more: see the guard in the generated cell.
"""
+ vram_source = vram_source or {}
encoded = {name: _encode_bytes(json.dumps(nb).encode("utf-8")) for name, nb in payloads.items()}
isolation = isolation or {}
system_site = {name: bool(isolation.get(name, True)) for name in payloads}
- order = list(payloads)
+ # Only legs carry overlays; Studio's halves install into their own tree.
+ overlay_specs = {
+ name: tuple(specs) for name, specs in (overlays or {}).items() if specs and name in payloads
+ }
+ # The CARD queue, which is not every payload. cpu_lane runs beside the
+ # cards and after_gpu runs once they are free; leaving either in here would
+ # hand it a card and defeat the point of both.
+ off_queue = {name for name in (cpu_lane, after_gpu) if name}
+ for name in off_queue:
+ if name not in payloads:
+ raise KeyError(f"{name!r} is scheduled off the card queue but is not a payload")
+ order = [name for name in payloads if name not in off_queue]
+ if not order:
+ raise ValueError("every payload is off the card queue, so nothing would use a GPU")
+ # hf_home is left unset ON PURPOSE. See build_driver's docstring: a private
+ # root here downloads 12 GB into a cache no leg reads and reports success.
+ # Per-payload VRAM for the admission check. Off-queue payloads are absent
+ # on purpose: neither takes a card, so neither has a budget to consume.
+ vram_gb = {name: float(getattr(leg, "vram_gb", 1.0)) for name, leg in vram_source.items()}
+ # Studio's halves are not legs and have no Leg record. The test half is the
+ # only one that takes a card, and it is priced from what it loads: a 2B
+ # chat model at UD-Q4_K_XL plus a 0.5B LoRA. Deliberately generous -- an
+ # under-price here is what would let it share with something that does not
+ # fit, and the whole admission check is only as honest as its inputs.
+ for name in (cpu_lane, after_gpu):
+ if name:
+ vram_gb.setdefault(name, 2.2)
+ prefetch_blob = ""
+ if prefetch_repos:
+ prefetch_blob = _encode_bytes(
+ _prefetch_builder().prefetch_cell(list(prefetch_repos)).encode("utf-8")
+ )
setup = f"""import base64, gzip, json, os, pathlib, subprocess, sys, threading, time
print("{DRIVER_SENTINEL} start", flush=True)
WORK = pathlib.Path("/kaggle/working")
+
+# Virtualenvs do NOT live under WORK. /kaggle/working is 19.5 GB and is also
+# what Kaggle ships home as the artifact; $HOME and /tmp share a ~1 TB overlay.
+# Each leg's venv carries its own torch and its own NVIDIA runtime, and with
+# two legs per card there can now be FOUR of them alive at once -- which does
+# not fit in 19.5 GB and would surface as an install failing halfway through
+# for reasons that look nothing like a disk.
+#
+# Evidence stays on WORK, because that is the only thing that has to come back.
+VENV_ROOT = pathlib.Path("/tmp/t4ci_venvs")
+try:
+ VENV_ROOT.mkdir(parents=True, exist_ok=True)
+except OSError:
+ # A box without a writable /tmp is not one this kernel can fix. Falling
+ # back to WORK keeps a one-leg-per-card run working, which is the shape
+ # that fitted there before co-scheduling.
+ VENV_ROOT = WORK
+print("{DRIVER_SENTINEL}_VENV_ROOT " + json.dumps({{"path": str(VENV_ROOT)}}), flush=True)
+# Defined HERE, not beside the build below. run_one exports it into every
+# payload's env, and the Studio CPU lane calls run_one before the build block
+# is reached -- so assigning it there is a NameError on the first payload to
+# start, which is the one that runs earliest and would take the whole kernel
+# down with it.
+WHEEL_DIR = VENV_ROOT / "wheels"
PAYLOADS = {json.dumps(encoded)}
# Which payloads may see the Kaggle image's site-packages. A leg that
# replaces torch must not: pip would then treat torch's pinned NVIDIA
@@ -532,6 +688,18 @@ PAYLOADS = {json.dumps(encoded)}
# _defines is what caught that, which is the whole reason it exists.
SYSTEM_SITE = {system_site!r}
+# Per-leg pure-Python version pins, laid OVER the venv rather than resolved
+# into it. See Leg.overlay. Empty for a leg that wants the base as it resolved.
+OVERLAYS = {overlay_specs!r}
+# Distributions an overlay may never carry. torch, triton and bitsandbytes are
+# native and coupled to the driver stack: a shadowed copy on PYTHONPATH either
+# fails to import or, worse, imports and silently disagrees with the CUDA
+# runtime already loaded. The overlay exists to move transformers/trl/peft, not
+# to rebuild the machine. Substring match, lowercased, so `nvidia-cublas-cu12`
+# and `torchvision` are caught without listing every wheel NVIDIA ships.
+OVERLAY_DENY = ("torch", "triton", "bitsandbytes", "nvidia", "cuda",
+ "xformers", "flash")
+
# How many GPUs did we actually get? Assert rather than assume: a 1-GPU
# allocation would silently serialise everything onto device 0 and double
# every runtime while looking like a slow but healthy run.
@@ -551,6 +719,72 @@ N_GPU = len(GPUS)
# legs.KERNELS, so the packing decision lives beside the legs it packs.
ORDER = {order!r}
+# Requirement specs that EVERY leg installs identically, built once up front.
+# Derived from the legs' own install groups rather than written out again here,
+# because a second copy of the two SHAs is a second thing to keep in step -- and
+# one that drifts silently, since a wheel built from the wrong ref installs
+# perfectly.
+SHARED_WHEEL_SPECS = {shared_wheel_specs!r}
+
+# Payloads that are NOT in the card queue.
+#
+# CPU_LANE runs from t=0 alongside the training legs and never takes a card.
+# AFTER_GPU runs once the queue has drained and wants every card. Both are None
+# on an ordinary notebook-only kernel, which leaves the schedule exactly as it
+# was.
+CPU_LANE = {cpu_lane!r}
+AFTER_GPU = {after_gpu!r}
+# ALL_CARD payloads want EVERY card visible rather than one. They are not in
+# the card queue either, because the queue hands out one card at a time and
+# there is no index to give them.
+#
+# They are NOT the same as AFTER_GPU. AFTER_GPU is unpinned AND waits for the
+# queue to drain; an ALL_CARD leg reserves its (small) share on every card
+# through _admit_all and then runs BESIDE the training legs, which is what
+# makes covering unsloth's DEVICE_COUNT > 1 branch cost nothing.
+ALL_CARD = {all_card!r}
+# When true, AFTER_GPU does not wait for the cards to drain. It is admitted to
+# a card by the same VRAM check the legs use, so it runs BESIDE a light leg
+# rather than behind the last one.
+#
+# Worth ~211s in simulation, because Studio's install-then-test chain is the
+# critical path while the legs have slack. What it costs is real and is why it
+# is a flag and not the default: Studio then sees ONE card instead of two, and
+# its own device selection is part of what it exists to prove.
+AFTER_GPU_CONCURRENT = {after_gpu_concurrent!r}
+
+# Measured peak_reserved_gb per payload (legs.Leg.vram_gb), and the budget a
+# single card will admit. A T4 reports 14.56 GB usable; the budget is 13.0 so
+# that fragmentation and the allocator's own headroom are not the thing that
+# discovers the limit.
+#
+# This is what lets two legs share a card. On run 32611343797 the three Qwen
+# legs peaked at 0.70 GB EACH -- 4.8% of the card -- while gptoss peaked at
+# 12.78 GB, so "one payload per card" was leaving a card 95% empty for three
+# quarters of the run. gptoss still never shares: 12.78 + 0.70 = 13.48 is over
+# budget, which is the arithmetic rather than a special case.
+#
+# MAX_LEGS_PER_CARD caps it at 2 even where VRAM would allow 3, because VRAM is
+# not the scarce resource here. A Kaggle session has 4 vCPUs and a leg is ~87%
+# install, import and download; the legs contend for CORES long before they
+# contend for memory, and a third concurrent install buys nothing the profile
+# says is available.
+VRAM_GB = {vram_gb!r}
+CARD_VRAM_BUDGET_GB = 13.0
+# 1 when the venvs had to fall back to WORK. The fallback above says it "keeps
+# a one-leg-per-card run working", and until this line that was a description
+# of an intention rather than of anything the code did: MAX_LEGS_PER_CARD was a
+# constant, so a box with no writable /tmp would still build four torch venvs,
+# two at a time per card, in the 19.5 GB /kaggle/working that the comment at
+# the top of this block says they do not fit in. It would have surfaced as an
+# install failing halfway through for reasons that look nothing like a disk.
+MAX_LEGS_PER_CARD = 2 if VENV_ROOT != WORK else 1
+
+# The model prefetch, gzip+base64 like the payloads so its quoting survives
+# being embedded in a generated cell. Empty string means no prefetch, which is
+# what every kernel did before this existed.
+PREFETCH_BLOB = {prefetch_blob!r}
+
# A shortfall is INFRASTRUCTURE, and it has to be called that HERE, before a
# thread starts. `max(1, ...)` used to make one card look like enough: both
# payloads were pinned to device 0, each child still saw exactly one GPU and
@@ -584,7 +818,7 @@ lock = threading.Lock()
def _make_venv(idx, system_site):
"""Give a child its own site-packages. See this file's module docstring."""
- vdir = WORK / f"venv_{{idx}}"
+ vdir = VENV_ROOT / f"venv_{{idx}}"
try:
uv = subprocess.run(["which", "uv"], capture_output=True,
text=True).stdout.strip()
@@ -592,7 +826,23 @@ def _make_venv(idx, system_site):
subprocess.run([sys.executable, "-m", "pip", "install", "-q", "uv"],
check=True, timeout=900)
uv = "uv"
- venv_cmd = [uv, "venv", str(vdir), "--seed"]
+ # PIN THE INTERPRETER TO THE DRIVER'S OWN, and it is not cosmetic.
+ #
+ # uv's default python-preference is `managed`, so once ANY managed
+ # CPython exists under ~/.local/share/uv/python -- which something in a
+ # longer kernel eventually downloads -- a bare `uv venv` silently builds
+ # on THAT interpreter instead of the image's. `--system-site-packages`
+ # is still honoured and still inherits nothing, because a 3.13 venv
+ # cannot see a 3.12 site-packages.
+ #
+ # Measured, 6 sessions, deterministic per arm: the Default leg came up
+ # on python 3.13.13 / torch 2.12.1+cu130 / datasets 5.0.1 in all three
+ # arm-B runs and on 3.12.13 / torch 2.10.0+cu128 / datasets 4.3.0 in
+ # all three arm-A runs, from one commit. Every other leg stayed on
+ # 3.12. The 3.13 runs then failed 3/3 in `Dataset.from_dict` -- see the
+ # overlay comment below for why -- and the whole leg was resolving a
+ # stack no Kaggle user has.
+ venv_cmd = [uv, "venv", str(vdir), "--seed", "--python", sys.executable]
if system_site:
venv_cmd.append("--system-site-packages")
subprocess.run(venv_cmd, check=True, timeout=900)
@@ -602,8 +852,16 @@ def _make_venv(idx, system_site):
kname = f"t4ci{{idx}}"
subprocess.run([str(py), "-m", "ipykernel", "install", "--user",
"--name", kname], check=True, timeout=600)
+ # The version is REPORTED, not assumed. A venv on the wrong interpreter
+ # is invisible in the driver log otherwise, and it was: three sessions
+ # went red before anyone read a python version out of a leg's report.
+ _ver = subprocess.run(
+ [str(py), "-c", "import sys;print('%d.%d.%d' % sys.version_info[:3])"],
+ capture_output=True, text=True, timeout=120).stdout.strip()
print(f"{DRIVER_SENTINEL}_VENV " + json.dumps(
- {{"idx": idx, "kernel": kname, "system_site": system_site}}),
+ {{"idx": idx, "kernel": kname, "system_site": system_site,
+ "python": _ver,
+ "driver_python": "%d.%d.%d" % sys.version_info[:3]}}),
flush=True)
return str(py), kname, str(vdir / "bin")
except Exception as exc:
@@ -617,7 +875,17 @@ def run_one(name, gpu_index, idx):
env = dict(os.environ)
# One GPU per payload. This is what makes the run comparable to the
# single T4 a Colab user gets; two visible GPUs is a different test.
- env["CUDA_VISIBLE_DEVICES"] = str(gpu_index)
+ #
+ # gpu_index None means DO NOT PIN, and it is not the same as pinning to
+ # nothing. It is used by the Studio lanes: the install half must still see
+ # both cards, because install.sh --local resolves torch and a CPU-only
+ # torch resolved by an installer that could not find a device is the exact
+ # regression Studio's verify cell exists to catch; and the test half wants
+ # both because Studio's own device selection is part of what it tests, so
+ # masking one would test a machine nobody has. Blanking the variable would
+ # do the opposite of both.
+ if gpu_index is not None:
+ env["CUDA_VISIBLE_DEVICES"] = str(gpu_index)
env["PYTHONUNBUFFERED"] = "1"
# One compile cache per payload. `unsloth_compiled_cache` is a RELATIVE
# path resolved against the working directory, which both papermill
@@ -629,6 +897,41 @@ def run_one(name, gpu_index, idx):
env["UNSLOTH_COMPILE_LOCATION"] = str(
WORK / ("unsloth_compiled_cache_" + pathlib.Path(name).stem))
+ # The OTHER three caches, which a separate venv does not protect.
+ #
+ # UNSLOTH_COMPILE_LOCATION above covered unsloth's generated modules and
+ # was assumed to be the whole story. It is not. torch and triton key their
+ # caches off $TMPDIR, not off the interpreter, so on torch 2.9.1 an unset
+ # TORCHINDUCTOR_CACHE_DIR resolves to `tempfile.gettempdir()/torchinductor_
+ # $USER` (torch/_inductor/runtime/cache_dir_utils.py:22) and the triton
+ # cache lands UNDER that same directory. Every leg on the box therefore
+ # shared /tmp/torchinductor_root -- four payloads whose entire purpose is
+ # to install DIFFERENT transformers/TRL/peft versions and compile the same
+ # modules, writing one cache between them.
+ #
+ # Per-leg directories rather than TORCHINDUCTOR_FORCE_DISABLE_CACHES or
+ # TRITON_ALWAYS_COMPILE, which are the blunter knobs and do exist
+ # (torch/compiler/config.py:85 and triton/knobs.py:358). Those force every
+ # leg to recompile from cold, which spends time on the critical path this
+ # kernel is being shortened along, and makes the legs measure a machine no
+ # user has. Isolation is what is wanted here, not cache defeat.
+ #
+ # TMPDIR is set too, and it is the belt to the braces: anything else that
+ # resolves a cache through gettempdir() follows it without this file having
+ # to know the variable's name.
+ _leg_tmp = VENV_ROOT / ("tmp_" + pathlib.Path(name).stem)
+ for _sub in ("", "inductor", "triton"):
+ try:
+ (_leg_tmp / _sub).mkdir(parents=True, exist_ok=True)
+ except OSError:
+ pass
+ env["TMPDIR"] = str(_leg_tmp)
+ env["TORCHINDUCTOR_CACHE_DIR"] = str(_leg_tmp / "inductor")
+ env["TRITON_CACHE_DIR"] = str(_leg_tmp / "triton")
+ # Where the once-built wheels are. Read by the install cell, which falls
+ # back to its git specs when the directory is empty or absent.
+ env["UNSLOTH_CI_WHEEL_DIR"] = str(WHEEL_DIR)
+
made = _make_venv(idx, SYSTEM_SITE.get(name, True))
if not made:
# No venv means the SHARED system site-packages, which is the one
@@ -650,6 +953,90 @@ def run_one(name, gpu_index, idx):
env["VIRTUAL_ENV"] = str(pathlib.Path(bindir).parent)
env["UV_SYSTEM_PYTHON"] = "0"
+ # The per-leg OVERLAY: this leg's version pins laid over the venv rather
+ # than resolved into it.
+ #
+ # Two pip calls, and the first one is the point. `--dry-run --report`
+ # resolves the FULL closure against the environment as it actually is, so
+ # the second call can be `--no-deps` over a manifest instead of a guess. An
+ # overlay assembled by naming packages by hand pairs a new transformers with
+ # the base's older tokenizers or huggingface_hub, and fails minutes later
+ # inside a payload with an import error that reads like a code bug.
+ #
+ # Never fatal. A leg whose overlay fails runs on the base versions and says
+ # so: that is a leg testing the wrong versions, which the payload's own
+ # resolved-version record makes visible, and it is strictly better than
+ # standing the leg down and reporting nothing at all.
+ _ov_specs = OVERLAYS.get(name) or ()
+ if _ov_specs:
+ # The trailing `site-packages` is LOAD-BEARING, and it is the one part
+ # of this path that looks like decoration.
+ #
+ # dill decides whether to pickle a module by REFERENCE or BY VALUE with
+ # `_is_builtin_module` (dill/_dill.py), which answers yes only if the
+ # module's __file__ starts with a sys prefix, ends with an extension
+ # suffix, or contains the literal string 'site-packages'. A plain
+ # `pip install --target /tmp/t4ci_venvs/overlay_X` satisfies none of
+ # those, so every package in the overlay is pickled by value.
+ #
+ # That matters because `Dataset.from_dict` fingerprints through dill:
+ # `datasets/utils/_dill.py:_save_arrowTable` saves `create_arrowTable`,
+ # dill walks its globals, reaches the pyarrow MODULE, pickles it by
+ # value, and hits pyarrow's Cython `MonthDayNano`, whose __module__ is
+ # `builtins`. The result is
+ # PicklingError: Can't pickle <class 'MonthDayNano'>:
+ # it's not found as builtins.MonthDayNano
+ # from a two-line `Dataset.from_dict` on a dict of plain strings, with
+ # nothing in the traceback that names this driver.
+ #
+ # Reproduced on CPU in seconds against a byte-identical package tree,
+ # with the DIRECTORY NAME as the only variable: the plain --target dir
+ # raised, the copy named site-packages returned a fingerprint.
+ _ov_dir = VENV_ROOT / ("overlay_" + pathlib.Path(name).stem) / "site-packages"
+ _ov_report = _leg_tmp / "overlay_report.json"
+ _ov_t0 = time.time()
+ _ov_rec = {{"payload": name, "requested": list(_ov_specs)}}
+ try:
+ _rc = subprocess.run(
+ [py, "-m", "pip", "install", "--dry-run", "--quiet",
+ "--report", str(_ov_report)] + list(_ov_specs),
+ capture_output=True, text=True, timeout=600)
+ _closure = []
+ if _ov_report.exists():
+ for _item in json.loads(_ov_report.read_text()).get("install", []):
+ _meta = _item.get("metadata", {{}})
+ _closure.append((_meta.get("name", "?"), _meta.get("version", "?")))
+ _keep, _dropped = [], []
+ for _nm, _ver in _closure:
+ if any(_d in _nm.lower() for _d in OVERLAY_DENY):
+ _dropped.append(f"{{_nm}}=={{_ver}}")
+ else:
+ _keep.append(f"{{_nm}}=={{_ver}}")
+ _ov_rec.update({{"resolved": _keep, "denied": _dropped,
+ "resolve_returncode": _rc.returncode}})
+ if _keep:
+ _rc2 = subprocess.run(
+ [py, "-m", "pip", "install", "--quiet", "--no-deps",
+ "--target", str(_ov_dir)] + _keep,
+ capture_output=True, text=True, timeout=900)
+ _ov_rec["install_returncode"] = _rc2.returncode
+ if _rc2.returncode == 0:
+ # PREPENDED, so the overlay wins over the venv. The payload
+ # child reads this at interpreter start, which is the only
+ # moment it can matter: an already-imported transformers
+ # cannot be displaced by any later sys.path edit.
+ env["PYTHONPATH"] = str(_ov_dir) + os.pathsep + env.get("PYTHONPATH", "")
+ _ov_rec["active"] = True
+ else:
+ _ov_rec["error"] = _rc2.stderr[-400:]
+ else:
+ _ov_rec["error"] = "resolved to nothing installable"
+ except BaseException as _exc: # noqa: BLE001
+ _ov_rec["error"] = f"{{type(_exc).__name__}}: {{_exc}}"[:300]
+ _ov_rec["seconds"] = round(time.time() - _ov_t0, 1)
+ _ov_rec.setdefault("active", False)
+ print(f"{DRIVER_SENTINEL}_OVERLAY " + json.dumps(_ov_rec), flush=True)
+
log = WORK / (pathlib.Path(name).stem + "_driver.log")
started = time.time()
rc, err = None, ""
@@ -680,7 +1067,7 @@ def run_one(name, gpu_index, idx):
# log are the evidence and they are NOT touched.
try:
import shutil as _sh
- _sh.rmtree(WORK / f"venv_{{idx}}", ignore_errors=True)
+ _sh.rmtree(VENV_ROOT / f"venv_{{idx}}", ignore_errors=True)
except Exception:
pass
print(f"{DRIVER_SENTINEL}_DONE " + json.dumps({{name: results[name]}}),
@@ -699,7 +1086,11 @@ def run_one(name, gpu_index, idx):
# (t4ci{{idx}}), which must stay distinct per PAYLOAD or two legs would share
# an interpreter and the differing library sets this whole file exists to keep
# apart would land in one tree.
-_queue = list(enumerate(ORDER))
+# ALL_CARD legs are filtered out here rather than left out of ORDER, so ORDER
+# stays the single declaration of what this kernel runs and the report's
+# ordering is unchanged. Their INDEX is preserved for run_one's numbering.
+_queue = [(i, n) for i, n in enumerate(ORDER) if n not in ALL_CARD]
+_all_card_queue = [(i, n) for i, n in enumerate(ORDER) if n in ALL_CARD]
# Each card's FIRST leg is assigned here, by position, and not raced for. A
# plain shared queue looked equivalent and was not: with payloads that finish
# quickly -- stubs, or an install that dies in seconds -- card 0 could claim,
@@ -711,8 +1102,119 @@ _queue = list(enumerate(ORDER))
# to run.
SEEDS = _queue[:N_GPU]
pending = _queue[N_GPU:]
+# Reserved BEFORE any worker thread exists, further down. A seat promised to a
+# seed but not taken until that worker wakes is a seat a free worker can see as
+# empty, which is exactly how run 32667451396 put gptoss and control on one
+# card. See the _admit reservation loop below the definition.
pending_lock = threading.Lock()
+# What each card is currently carrying, in GB and in count. Admission is taken
+# under the same lock as the queue so two workers cannot both look at a card
+# with room and both decide to use it.
+card_load = {{i: 0.0 for i in range(N_GPU)}}
+card_count = {{i: 0 for i in range(N_GPU)}}
+# Concurrent payloads that hold no card index. A list rather than an int so the
+# nested lane functions can mutate it without a `global` in generated source.
+unpinned_count = [0]
+
+
+def _admit(gpu_index, name):
+ """Reserve room for `name` on `gpu_index`, or return False.
+
+ Called holding `pending_lock`. Returning False leaves the leg on the queue
+ for a card that can take it -- or for this one, later, once something
+ finishes. It must never mutate on the failing path or a refused leg would
+ leak capacity it never used.
+ """
+ want = VRAM_GB.get(name, 1.0)
+ if card_count[gpu_index] >= MAX_LEGS_PER_CARD:
+ return False
+ # An unpinned payload contends for the same 4 vCPUs without holding a card,
+ # so the total bound has to see it here as well. Enforcing it only in
+ # _admit_all would let the card queue fill up beside it and put N_GPU *
+ # MAX_LEGS_PER_CARD + 1 installs on four cores.
+ if sum(card_count.values()) + unpinned_count[0] >= N_GPU * MAX_LEGS_PER_CARD:
+ return False
+ if card_count[gpu_index] and card_load[gpu_index] + want > CARD_VRAM_BUDGET_GB:
+ return False
+ card_load[gpu_index] += want
+ card_count[gpu_index] += 1
+ return True
+
+
+def _release(gpu_index, name):
+ with pending_lock:
+ card_load[gpu_index] -= VRAM_GB.get(name, 1.0)
+ card_count[gpu_index] -= 1
+
+
+def _admit_all(name):
+ """Reserve room for `name` on EVERY card, or on none of them.
+
+ All or nothing, under one hold of `pending_lock`. A partial reservation is
+ the failure worth naming: the leg would hold a seat on card 0 while waiting
+ for card 1, which is capacity nothing is using and which can deadlock
+ against a big leg waiting for card 0.
+
+ VRAM is charged to EVERY card and the CONTENTION to a global counter, and
+ both halves of that are measurements rather than preferences. `card_load` tracks memory, and an all-card leg genuinely holds
+ a CUDA context on each card -- measured at 1.2 GB on
+ unsloth-probe-multigpu-r2-a280e2, which is why its declaration is 1.2 rather
+ than the 0.7 the other Qwen legs carry. `card_count` tracks something else
+ entirely: MAX_LEGS_PER_CARD exists as a proxy for 4-vCPU CONTENTION, and its
+ own comment says so -- "the legs contend for CORES long before they contend
+ for memory". An all-card leg is ONE process. Charging it a slot on both
+ cards counts one process twice.
+
+ Charging the count PER CARD is wrong in both available ways, and each was
+ caught by a different instrument.
+
+ Charged on EVERY card it takes one of the two slots on each, and the driver
+ simulation showed no card ever holding two legs at once -- the co-tenancy
+ that makes a fifth and sixth leg affordable, gone.
+
+ Charged on ONE card it decides a placement it has nothing to do with, and
+ that cost 188.7s of wall clock on hardware. The A/B of
+ unsloth-probe-ab-baseline-5leg-20db9c against
+ unsloth-probe-ab-with-multigpu-6169ca, same commit and same day:
+
+ A 1936.4s gpu0 10.0s idle gpu1 3.0s idle
+ B 2125.1s gpu0 622.6s idle gpu1 0.0s idle
+
+ In B, Studio wanted a card at t=653. gpu0 held canary plus this leg's count
+ -- two, the cap -- so it was refused and took gpu1, where the 1707s vision
+ leg had ~1080s left. gpu1 then carried BOTH of the two longest payloads
+ while gpu0 idled for over ten minutes. The leg itself ran 28.2 -> 668.1
+ fully overlapped and contributed nothing directly.
+
+ So the contention goes on a GLOBAL counter. MAX_LEGS_PER_CARD is a proxy
+ for 4 vCPUs, and its own comment says so -- "the legs contend for CORES long
+ before they contend for memory". An unpinned process contends for those
+ cores without occupying a card, and the total concurrency bound
+ (N_GPU * MAX_LEGS_PER_CARD) is what it should count against. VRAM still goes
+ on every card, because the CUDA context really is on every card.
+ """
+ want = VRAM_GB.get(name, 1.0)
+ with pending_lock:
+ for g in range(N_GPU):
+ if card_count[g] and card_load[g] + want > CARD_VRAM_BUDGET_GB:
+ return False
+ if sum(card_count.values()) + unpinned_count[0] >= N_GPU * MAX_LEGS_PER_CARD:
+ return False
+ for g in range(N_GPU):
+ card_load[g] += want
+ unpinned_count[0] += 1
+ return True
+
+
+def _release_all(name):
+ want = VRAM_GB.get(name, 1.0)
+ with pending_lock:
+ for g in range(N_GPU):
+ card_load[g] -= want
+ unpinned_count[0] -= 1
+
+
def worker(gpu_index, seed):
if seed is not None:
# The 5s stagger the per-payload threads used to get from the start
@@ -723,22 +1225,253 @@ def worker(gpu_index, seed):
if gpu_index:
time.sleep(5 * gpu_index)
idx, name = seed
- run_one(name, gpu_index, idx)
+ # No _admit here. The seat was taken in the main thread before ANY
+ # worker started, and taking it here instead is what let gptoss onto
+ # a card that a seed had already been promised: the stagger below
+ # meant card 1 sat unreserved for 5s, a free worker looked at an
+ # apparently empty card, and 12.78 + 0.70 GB then ran on one 13.0 GB
+ # budget for 691s on run 32667451396. The old call also discarded the
+ # answer, so even once it did refuse, the leg ran anyway.
+ try:
+ run_one(name, gpu_index, idx)
+ finally:
+ _release(gpu_index, name)
while True:
+ picked = None
with pending_lock:
if not pending:
+ # Nothing left to start. A worker whose card still holds a leg
+ # simply exits; the OTHER thread on that card is the one still
+ # running it, and the joins below wait for that.
return
- idx, name = pending.pop(0)
- run_one(name, gpu_index, idx)
+ # First leg on the queue this card can actually hold. Not strictly
+ # the head: gptoss cannot join an occupied card, and refusing to
+ # look past it would idle a card that has room for the small leg
+ # behind it.
+ for position, (idx, name) in enumerate(pending):
+ if _admit(gpu_index, name):
+ picked = pending.pop(position)
+ break
+ if picked is None:
+ # Every remaining leg is too big for what this card has free right
+ # now. Wait for a release rather than spin; something is running or
+ # the queue would have been empty.
+ time.sleep(1.0)
+ continue
+ idx, name = picked
+ try:
+ run_one(name, gpu_index, idx)
+ finally:
+ _release(gpu_index, name)
+
+# The CPU lane, started BEFORE the cards are handed out and running beside
+# them. It carries work that needs no GPU -- the Studio install: a checkout, a
+# frontend build, a venv, a llama.cpp download and a Playwright browser -- so
+# every second of it that overlaps the training legs is a second nobody waits
+# for. It is not given a card and does not queue for one.
+# The PREFETCH lane. Started before anything else, because it is the only lane
+# whose value is entirely a function of how early it starts: a prefetch pays
+# only for what it finishes before the leg that wants the model begins, and
+# legs.KERNELS puts gptoss third specifically to give this lane that window.
+#
+# A driver THREAD rather than a papermill payload, deliberately. It needs the
+# image's huggingface_hub and nothing else -- no venv, no install, no card --
+# so as a payload it would spend its first ~60s building a virtualenv it has no
+# use for, which is 60s taken off the head start that is the whole point.
+prefetch_thread = None
+if PREFETCH_BLOB:
+ def _prefetch_lane():
+ src = gzip.decompress(base64.b64decode(PREFETCH_BLOB)).decode("utf-8")
+ # The cell never raises by contract, but exec'ing generated source on a
+ # lane nobody joins before the cards start is not the place to find out
+ # otherwise: an escape here would kill the thread silently and the run
+ # would look like a prefetch that simply found nothing to do.
+ try:
+ exec(compile(src, "<prefetch>", "exec"), {{"__name__": "prefetch"}})
+ except BaseException as exc:
+ print("{DRIVER_SENTINEL}_PREFETCH_FAILED " + json.dumps(
+ {{"error": f"{{type(exc).__name__}}: {{exc}}"}}), flush=True)
+ prefetch_thread = threading.Thread(target = _prefetch_lane, daemon = True)
+ prefetch_thread.start()
+ print("{DRIVER_SENTINEL}_PREFETCH " + json.dumps({{"started": True}}), flush=True)
+
+cpu_thread = None
+if CPU_LANE:
+ def _cpu_lane():
+ run_one(CPU_LANE, None, len(ORDER))
+ cpu_thread = threading.Thread(target = _cpu_lane, daemon = False)
+ cpu_thread.start()
+ print("{DRIVER_SENTINEL}_CPU_LANE " + json.dumps({{"payload": CPU_LANE}}), flush=True)
+
+# Build unsloth and unsloth_zoo ONCE, before any leg starts.
+#
+# Every leg's first two install groups are byte-identical -- the same two
+# pinned SHAs, zoo from main and unsloth from the ref under test -- and pip
+# does not cache a VCS build, so run 32679427416 cloned and built both of them
+# FOUR times. Install was 149-191s per leg there, the single largest phase of
+# every leg and 41% of gpt-oss.
+#
+# Synchronous, and that is the whole design. A background lane would lose the
+# race it exists to win: the legs are admitted to their cards and start
+# installing at t=0, so a wheel that is still building when they get there
+# saves nothing. Studio's CPU lane is started FIRST so its 345s overlaps this
+# rather than queueing behind it.
+#
+# Building from the same SHAs the groups name is what keeps this honest: the
+# wheel IS main-plus-the-PR, not a substitute for it. A leg whose wheel is
+# missing falls back to its original git spec, so a failed build costs the time
+# it took and changes nothing about what is tested.
+if SHARED_WHEEL_SPECS:
+ _t0 = time.time()
+ try:
+ WHEEL_DIR.mkdir(parents=True, exist_ok=True)
+ _proc = subprocess.run(
+ [sys.executable, "-m", "pip", "wheel", "--no-deps", "-q",
+ "-w", str(WHEEL_DIR), *SHARED_WHEEL_SPECS],
+ capture_output=True, text=True, timeout=900)
+ _built = sorted(p.name for p in WHEEL_DIR.glob("*.whl"))
+ print("{DRIVER_SENTINEL}_WHEELS " + json.dumps({{
+ "seconds": round(time.time() - _t0, 1),
+ "returncode": _proc.returncode,
+ "built": _built,
+ "error": None if _proc.returncode == 0 else _proc.stderr.strip()[-400:],
+ }}), flush=True)
+ except BaseException as _exc: # noqa: BLE001
+ # Never fatal. The legs still carry their git specs.
+ print("{DRIVER_SENTINEL}_WHEELS " + json.dumps({{
+ "seconds": round(time.time() - _t0, 1), "returncode": None,
+ "built": [], "error": f"{{type(_exc).__name__}}: {{_exc}}",
+ }}), flush=True)
+
+# MAX_LEGS_PER_CARD workers per card, not one. The extra worker is what lets a
+# card pick up a second small leg beside the one it is already running; it
+# takes nothing when VRAM says no, so a card carrying gptoss behaves exactly as
+# it did when there was one worker each.
+# Take every seed's seat before a single worker exists. Each seed is the first
+# leg on an empty card, so each of these always succeeds; the point is WHEN it
+# happens, not whether. Doing it inside the seed worker left card 1 unreserved
+# for the length of its start stagger.
+for _g, _seed in enumerate(SEEDS):
+ _admit(_g, _seed[1])
+print("{DRIVER_SENTINEL}_SEEDS " + json.dumps(
+ {{str(_g): _seed[1] for _g, _seed in enumerate(SEEDS)}}), flush=True)
+
+# The ALL_CARD lane(s). One thread each, waiting for room on every card at
+# once, then running UNPINNED so torch reports every device.
+#
+# Started before the card workers so the leg is queued for the first moment
+# both cards have a free slot, rather than after whatever the workers grab in
+# the same instant. It still cannot jump the seeds: those seats were taken
+# above, before any thread existed.
+all_card_threads = []
+for _idx, _name in _all_card_queue:
+ def _all_card_lane(idx=_idx, name=_name):
+ while not _admit_all(name):
+ # Everything is full. Something is running or the queue would be
+ # empty, so wait for a release rather than spin.
+ time.sleep(1.0)
+ try:
+ print("{DRIVER_SENTINEL}_ALL_CARD " + json.dumps(
+ {{"payload": name, "gpus": N_GPU}}), flush=True)
+ # gpu_index None means DO NOT PIN, which is the whole point: an
+ # index here would set CUDA_VISIBLE_DEVICES and the leg would
+ # measure the single-card branch under a multi-GPU name.
+ run_one(name, None, idx)
+ finally:
+ _release_all(name)
+ _t = threading.Thread(target = _all_card_lane, daemon = False)
+ _t.start()
+ all_card_threads.append(_t)
threads = []
for gpu_index in range(N_GPU):
- seed = SEEDS[gpu_index] if gpu_index < len(SEEDS) else None
- t = threading.Thread(target=worker, args=(gpu_index, seed), daemon=False)
- t.start()
- threads.append(t)
+ for slot in range(MAX_LEGS_PER_CARD):
+ seed = SEEDS[gpu_index] if (slot == 0 and gpu_index < len(SEEDS)) else None
+ t = threading.Thread(target=worker, args=(gpu_index, seed), daemon=False)
+ t.start()
+ threads.append(t)
+def _after_gpu_blocked():
+ """The install half must have SUCCEEDED before the test half is worth
+ running: it is what puts the interpreter and the llama.cpp on disk, so
+ without it this half dies on a missing venv and reports that as a Studio
+ regression."""
+ prior = results.get(CPU_LANE) if CPU_LANE else None
+ return bool(CPU_LANE) and (prior is None or prior.get("returncode") != 0)
+
+
+def _after_gpu_skip():
+ results[AFTER_GPU] = {{
+ "returncode": None, "gpu": None, "seconds": 0.0,
+ "error": "the install lane did not succeed, so there is nothing "
+ "installed to test",
+ "kernel": None, "output_exists": False,
+ }}
+ print("{DRIVER_SENTINEL}_SKIPPED " + json.dumps({{AFTER_GPU: results[AFTER_GPU]}}),
+ flush=True)
+
+
+# Concurrent placement: AFTER_GPU takes a card as soon as its install lane is
+# done and some card has the VRAM, rather than waiting for the whole queue.
+after_thread = None
+if AFTER_GPU and AFTER_GPU_CONCURRENT:
+ def _after_gpu_lane():
+ if cpu_thread is not None:
+ cpu_thread.join()
+ if _after_gpu_blocked():
+ _after_gpu_skip()
+ return
+ chosen = None
+ while chosen is None:
+ with pending_lock:
+ for g in range(N_GPU):
+ if _admit(g, AFTER_GPU):
+ chosen = g
+ break
+ if chosen is None:
+ # Every card is full. Something is running or the legs would
+ # have finished, so wait for a release rather than spin.
+ time.sleep(1.0)
+ try:
+ print("{DRIVER_SENTINEL}_AFTER_GPU_SHARED " + json.dumps(
+ {{"payload": AFTER_GPU, "gpu": chosen}}), flush=True)
+ run_one(AFTER_GPU, chosen, len(ORDER) + 1)
+ finally:
+ _release(chosen, AFTER_GPU)
+ after_thread = threading.Thread(target = _after_gpu_lane, daemon = False)
+ after_thread.start()
+
for t in threads:
t.join()
+# Joined with the card workers, not after AFTER_GPU: an all-card leg runs
+# beside the training legs and must be finished before anything concludes the
+# cards are free.
+for t in all_card_threads:
+ t.join()
+
+# On the NON-concurrent path AFTER_GPU wants every card, so it waits for the
+# queue to drain. Studio's driver keeps both T4s visible on purpose -- "Studio's
+# own device selection is part of what is under test; masking one would test a
+# machine nobody has" -- and that stays true here.
+if cpu_thread is not None:
+ cpu_thread.join()
+if after_thread is not None:
+ after_thread.join()
+
+if AFTER_GPU and not AFTER_GPU_CONCURRENT:
+ if _after_gpu_blocked():
+ # Do NOT run it. The install half is what puts the interpreter and the
+ # llama.cpp on disk, so without it this half fails on a missing venv
+ # and reports that as a Studio regression. Recording the skip keeps the
+ # cause attached to the effect.
+ results[AFTER_GPU] = {{
+ "returncode": None, "gpu": None, "seconds": 0.0,
+ "error": "the install lane did not succeed, so there is nothing "
+ "installed to test",
+ "kernel": None, "output_exists": False,
+ }}
+ _after_gpu_skip()
+ else:
+ run_one(AFTER_GPU, None, len(ORDER) + 1)
print("{DRIVER_SENTINEL}_RESULTS " + json.dumps(results), flush=True)
'''
@@ -793,6 +1526,67 @@ print("{DRIVER_SENTINEL} complete", flush=True)
}
+STUDIO_INSTALL_NOTEBOOK = "studio_install.ipynb"
+STUDIO_TEST_NOTEBOOK = "studio_test.ipynb"
+
+
+def _studio_builder():
+ """Load ``kaggle_studio_ci/build_kernel.py`` under a name of its own.
+
+ By PATH, not by import. That directory and this one BOTH contain a module
+ called ``build_kernel`` (and both contain a ``report`` too), and the test
+ suite puts both on ``sys.path``, so a plain ``import build_kernel`` resolves
+ to whichever reached ``sys.modules`` first -- which is decided by test
+ order, not by intent. That exact collision has already been paid for once
+ here: adding one ``sys.path.insert`` to a test took nine unrelated summary
+ tests down with it.
+ """
+ import importlib.util
+
+ path = Path(__file__).resolve().parents[1] / "kaggle_studio_ci" / "build_kernel.py"
+ spec = importlib.util.spec_from_file_location("kaggle_studio_ci__build_kernel", path)
+ if spec is None or spec.loader is None:
+ raise ImportError(f"cannot load the Studio kernel builder from {path}")
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+def _prefetch_builder():
+ """Load ``kaggle_prefetch.py`` by PATH, for the reason ``_studio_builder``
+ documents: two sibling directories both ship a ``build_kernel`` and a
+ ``report``, so plain imports here resolve by test order rather than intent.
+ """
+ import importlib.util
+
+ path = Path(__file__).resolve().parents[1] / "kaggle_prefetch.py"
+ spec = importlib.util.spec_from_file_location("kaggle_ci__prefetch", path)
+ if spec is None or spec.loader is None:
+ raise ImportError(f"cannot load the prefetch builder from {path}")
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+def studio_payloads(*, unsloth_ref: str, repo_url: str, payload_args: str) -> dict[str, dict]:
+ """The Studio payload, split into its GPU-free half and its GPU half."""
+ studio = _studio_builder()
+ return {
+ STUDIO_INSTALL_NOTEBOOK: studio.build_payload_notebook(
+ unsloth_ref = unsloth_ref,
+ repo_url = repo_url,
+ payload_args = payload_args,
+ phase = "install",
+ ),
+ STUDIO_TEST_NOTEBOOK: studio.build_payload_notebook(
+ unsloth_ref = unsloth_ref,
+ repo_url = repo_url,
+ payload_args = payload_args,
+ phase = "test",
+ ),
+ }
+
+
def build_kernel(
payload_dir: Path,
leg_names,
@@ -802,9 +1596,16 @@ def build_kernel(
extra_args: tuple[str, ...],
per_run_timeout: int,
skip_reference: bool = False,
+ studio: dict | None = None,
+ prefetch_repos: tuple[str, ...] = (),
+ after_gpu_concurrent: bool = False,
+ shared_wheels: bool = False,
) -> dict:
payloads = {}
isolation = {}
+ overlays: dict[str, tuple[str, ...]] = {}
+ legs_by_payload = {}
+ leg_groups: dict[str, list[list[str]]] = {}
for leg in resolve(leg_names):
name = f"t4_{leg.name}.ipynb"
payloads[name] = build_payload_notebook(
@@ -816,6 +1617,48 @@ def build_kernel(
reference = "" if skip_reference else None,
)
isolation[name] = leg.system_site_packages
+ overlays[name] = tuple(leg.overlay)
+ legs_by_payload[name] = leg
+ leg_groups[name] = expand_install(
+ leg,
+ unsloth_ref = unsloth_ref,
+ zoo_ref = zoo_ref,
+ payload_dir = payload_dir,
+ )
+ # The card queue is the LEGS. Studio's two halves ride the same kernel but
+ # not the same queue, so expected_gpus is derived before they are added:
+ # they are what the cards are freed FOR, not more work to schedule onto
+ # them.
+ expected_gpus = min(len(payloads), SESSION_GPUS)
+ # An all-card leg needs SESSION_GPUS whatever the payload count is. Without
+ # this a single-leg dispatch (`--legs multi_gpu`) derives 1, the shortfall
+ # guard is satisfied by a one-card allocation, and the run proceeds to fail
+ # inside the payload on `device_count() == 2` -- a capacity fact reported as
+ # a payload verdict, which is precisely the confusion the guard's own
+ # comment says it exists to prevent.
+ if any(leg.all_cards for leg in legs_by_payload.values()):
+ expected_gpus = SESSION_GPUS
+ # Opt in. On run 32689629906 the wheels helped the leg that runs ALONE and
+ # cost the three that run concurrently, and that run moved the caches at the
+ # same time, so the effect is not yet attributable to either. Off by default
+ # until one variable at a time says otherwise.
+ shared_wheel_specs = _shared_vcs_specs(leg_groups) if shared_wheels else ()
+ # Derived from the legs themselves rather than named again here: a second
+ # list of which legs want every card is a second thing to keep in step, and
+ # one that drifts silently -- a leg dropped from it simply gets pinned and
+ # its multi-GPU assertions go looking for a second card that is not there.
+ all_card = tuple(payload for payload, leg in legs_by_payload.items() if leg.all_cards)
+ cpu_lane = after_gpu = None
+ if studio:
+ payloads.update(studio_payloads(**studio))
+ cpu_lane = STUDIO_INSTALL_NOTEBOOK
+ after_gpu = STUDIO_TEST_NOTEBOOK
+ # Both halves see the Kaggle image, as the standalone Studio kernel
+ # does: install.sh builds its OWN venv under STUDIO_HOME and that is
+ # the interpreter every assertion runs under, so the outer one only has
+ # to be able to start papermill.
+ isolation[STUDIO_INSTALL_NOTEBOOK] = True
+ isolation[STUDIO_TEST_NOTEBOOK] = True
# min(), so a one-leg kernel (a --legs dispatch, or a debugging run) still
# stands down only on a genuinely empty allocation rather than demanding a
# second card it will never use.
@@ -823,10 +1666,48 @@ def build_kernel(
payloads,
per_run_timeout,
isolation,
- expected_gpus = min(len(payloads), SESSION_GPUS),
+ expected_gpus = expected_gpus,
+ cpu_lane = cpu_lane,
+ after_gpu = after_gpu,
+ prefetch_repos = prefetch_repos,
+ vram_source = legs_by_payload,
+ after_gpu_concurrent = after_gpu_concurrent,
+ shared_wheel_specs = shared_wheel_specs,
+ overlays = overlays,
+ all_card = all_card,
)
+def _shared_vcs_specs(leg_groups: dict[str, list[list[str]]]) -> tuple[str, ...]:
+ """VCS requirements that EVERY leg installs identically.
+
+ Derived rather than declared. The two SHAs already live in the install
+ groups, and a second copy here would be a second thing to keep in step --
+ one that drifts silently, because a wheel built from the wrong ref installs
+ perfectly and fails nothing.
+
+ "Every leg" is the point of the intersection. A spec only one leg carries is
+ part of what that leg is testing and must keep its own resolution; building
+ it once and sharing it would quietly make the legs agree about something
+ they exist to disagree about. Two legs pinning the SAME package to
+ DIFFERENT refs therefore yields nothing shared, which is the safe answer.
+ """
+ if not leg_groups:
+ return ()
+ sets = []
+ for groups in leg_groups.values():
+ specs = set()
+ for group in groups:
+ for token in group:
+ if "@ git+" in token or token.startswith("git+"):
+ specs.add(token)
+ sets.append(specs)
+ common = set.intersection(*sets) if sets else set()
+ # Sorted for a stable kernel: an unordered set would rebuild the notebook
+ # differently run to run and make a diff of two kernels unreadable.
+ return tuple(sorted(common))
+
+
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--payload-dir", required = True)
@@ -862,8 +1743,54 @@ def main() -> int:
help = "build with no band check at all. Only for the one run that recaptures a reference",
)
ap.add_argument("--per-run-timeout", type = int, default = 2400)
+ ap.add_argument(
+ "--shared-wheels",
+ action = "store_true",
+ help = "build unsloth and unsloth_zoo once up front and install every "
+ "leg from the wheel instead of resolving the git spec per leg",
+ )
+ ap.add_argument(
+ "--studio-concurrent",
+ action = "store_true",
+ help = "let the Studio GPU half share a card with a light training leg "
+ "instead of waiting for the queue to drain. Faster, but Studio then "
+ "sees one T4 rather than two, which narrows what it proves",
+ )
+ ap.add_argument(
+ "--no-prefetch",
+ action = "store_true",
+ help = "do not warm the HF cache on a background lane. The models are "
+ "then downloaded by the legs that want them, as they were before the "
+ "lane existed. Note that legs.KERNELS is ORDERED for the prefetch -- "
+ "gptoss sits third to give the lane a window -- so this flag is for "
+ "isolating the lane when debugging, not a supported way to run",
+ )
+ ap.add_argument(
+ "--with-studio",
+ action = "store_true",
+ help = "also carry the Studio GPU payload in this kernel, split in two: "
+ "its checkout/install/browser half runs on a CPU lane beside the "
+ "training legs and never takes a card, and its assertions run once "
+ "the legs have freed both. Only valid with --all-kernels",
+ )
+ ap.add_argument(
+ "--studio-repo-url",
+ default = "https://github.com/unslothai/unsloth",
+ help = "repository the Studio half checks out and installs",
+ )
+ ap.add_argument(
+ "--studio-args",
+ default = "",
+ help = "extra args for tests/kaggle/studio_gpu/run_studio_gpu.py",
+ )
args = ap.parse_args()
+ if args.with_studio and not args.all_kernels:
+ # --legs builds ONE kernel of named legs, which is the debugging shape.
+ # Attaching Studio to it would put a 10-minute install beside a
+ # deliberately narrowed run.
+ raise SystemExit("--with-studio requires --all-kernels")
+
if args.all_kernels == bool(args.legs):
raise SystemExit("pass exactly one of --legs and --all-kernels")
@@ -876,7 +1803,17 @@ def main() -> int:
if not plan[0]:
raise SystemExit("--legs named nothing")
- for names, out in zip(plan, outputs):
+ # Studio rides the FIRST kernel only. There is one kernel today, so this is
+ # not a choice with consequences yet, but naming it stops a future second
+ # kernel quietly installing Studio twice and paying for it twice.
+ for index, (names, out) in enumerate(zip(plan, outputs)):
+ studio = None
+ if args.with_studio and index == 0:
+ studio = {
+ "unsloth_ref": args.unsloth_ref,
+ "repo_url": args.studio_repo_url,
+ "payload_args": args.studio_args,
+ }
driver = build_kernel(
Path(args.payload_dir),
names,
@@ -885,18 +1822,49 @@ def main() -> int:
extra_args = tuple(args.smoke_args.split()),
per_run_timeout = args.per_run_timeout,
skip_reference = args.skip_reference,
+ studio = studio,
+ # Only the kernel that CARRIES gptoss should pay for its 12 GB.
+ # There is one kernel today, so this reads as "always", but naming
+ # it stops a future second kernel prefetching a model it will never
+ # load -- which would be pure network cost, on a lane whose entire
+ # justification is that it is free.
+ prefetch_repos = () if args.no_prefetch else PREFETCH_REPOS,
+ after_gpu_concurrent = args.studio_concurrent,
+ shared_wheels = args.shared_wheels,
)
out.parent.mkdir(parents = True, exist_ok = True)
out.write_text(json.dumps(driver, indent = 1), encoding = "utf-8")
+ # Says whether Studio is aboard, because a build that quietly stopped
+ # packing it looks exactly like one that never asked for it.
print(
f"wrote {out} ({out.stat().st_size / 1024:.0f} KB) packing "
f"{len(names)} leg(s): {', '.join(names)}"
+ + (" + the Studio install and test halves" if studio else "")
)
# The launcher needs one --notebook per kernel and the expected payload
# count; both follow from the plan, so they are emitted here rather than
# restated in the workflow.
+ #
+ # Studio counts as ONE payload, not two, and that holds on both paths.
+ # Its two notebooks are halves of one experiment: on a healthy run the
+ # install half emits no report at all and the test half emits the only
+ # one, and on a broken install the install half emits a failure report and
+ # the driver then SKIPS the test half. Either way the kernel produces
+ # exactly one `studio-gpu` report, so counting the install half would make
+ # every healthy run look like it lost one.
+ #
+ # Getting this wrong in the other direction is worse and is why it is
+ # derived rather than typed: a merged kernel that quietly stopped running
+ # Studio would still report the four legs and go green.
+ legs = sum(len(n) for n in plan)
+ expected_payloads = legs + (1 if args.with_studio else 0)
_github_output("notebooks", " ".join(f"--notebook {o}" for o in outputs))
- _github_output("payloads", str(sum(len(n) for n in plan)))
+ _github_output("payloads", str(expected_payloads))
+ # The launcher counts every report in the kernel; each REPORTER counts only
+ # the labels it owns. So the T4 reporter is told the leg count, not the
+ # payload count -- handing it 5 would have it treat a complete four-leg
+ # result as short by one and dump the kernel log under a healthy run.
+ _github_output("legs", str(legs))
return 0
diff --git a/.github/scripts/kaggle_t4_ci/legs.py b/.github/scripts/kaggle_t4_ci/legs.py
index 5e18a6269d6..3e9f6c42dec 100644
--- a/.github/scripts/kaggle_t4_ci/legs.py
+++ b/.github/scripts/kaggle_t4_ci/legs.py
@@ -81,6 +81,63 @@ class Leg:
"unsloth",
"unsloth_zoo",
)
+ # Measured peak_reserved_gb on a Tesla T4, from the executed payload
+ # notebooks of run 32611343797. NOT an estimate and not a request: the
+ # driver uses it to decide whether two legs may share a card.
+ #
+ # gptoss 12.78 of 14.56 frontier 0.70
+ # canary 0.70 control 0.70
+ #
+ # The three Qwen legs are 4.8% of the card each, which is why sharing is
+ # possible at all; gptoss is 88%, which is why it never shares. A leg whose
+ # real appetite grows past what it declares here is the failure mode, so
+ # `test_the_declared_vram_matches_what_the_legs_reported` checks these
+ # against the peaks in the evidence rather than trusting them.
+ vram_gb: float = 1.0
+ # Pure-Python version pins this leg wants OVER the shared base, installed
+ # into a per-leg directory and put on PYTHONPATH ahead of it.
+ #
+ # This is the mechanism Studio already ships: `.venv_t5_530/`,
+ # `.venv_t5_550/`, `.venv_t5_510/` in studio/backend/utils/
+ # transformers_version.py, installed --no-deps, prepended to sys.path and
+ # propagated to children through PYTHONPATH. Copied rather than reinvented.
+ #
+ # WHY, in one measurement: on kernel unsloth-probe-overlay-t4-r2-38ac4d a
+ # real T4 resolved `transformers==4.57.6 + trl~=0.22.0` against the ambient
+ # environment to THREE packages, 115.9 MB, installed in 10.0s -- against the
+ # 158-207s each leg spends on its own install groups today. The overlay is
+ # the version delta, not the environment.
+ #
+ # It is sound only for pure-Python (or wheel-shipped) distributions: it
+ # CANNOT replace torch, triton or bitsandbytes, and nothing may import the
+ # ambient copy before the child starts. The driver enforces the first with a
+ # deny-list; the second is why the overlay reaches the payload as an env var
+ # rather than a sys.path edit inside an already-running interpreter.
+ #
+ # Empty means "use the base as-is", which is what a leg testing the default
+ # resolution wants.
+ overlay: tuple[str, ...] = ()
+ # Distributions to REMOVE after the install groups run. Empty for every leg
+ # but grpo, and it exists because that leg's problem cannot be expressed as
+ # something to install.
+ #
+ # FlashInfer cannot LINK on the Kaggle image: it ships the runtime
+ # libcuda.so.1 but not the driver stub libcuda.so, so `-lcuda` fails after
+ # nvcc has already succeeded. Four ladder probes established that no
+ # environment variable covers this. Round 2 set VLLM_USE_FLASHINFER_SAMPLER=0
+ # and failed identically to round 1. Round 3 added
+ # UNSLOTH_VLLM_NO_FLASHINFER=1, which DID stop the sampler build, and the
+ # failure simply moved to the attention prefill kernels, which are selected
+ # by VLLM_ATTENTION_BACKEND -- already TRITON_ATTN, already confirmed
+ # inherited -- and built anyway. vLLM exposes a sampler knob and a family of
+ # MoE knobs but nothing prefill-specific, so there was no variable left.
+ #
+ # Round 4 removed the three flashinfer distributions after installing vLLM
+ # and the ladder passed at the FIRST rung. Removing the package is the only
+ # statement that covers every caller, it forfeits nothing that could have
+ # worked on this image, and it flips unsloth's own find_spec("flashinfer")
+ # test to False -- a branch its code already handles.
+ uninstall: tuple[str, ...] = ()
# Filename under <payload-dir>/references to band-check against, if any.
reference: str = ""
# Extra environment for the child process.
@@ -106,6 +163,25 @@ class Leg:
# resolve the whole stack, so it resolves a consistent one, at the cost of
# a download of a few minutes.
system_site_packages: bool = True
+ # Does this leg want EVERY card visible rather than being pinned to one?
+ #
+ # False for every leg but `multi_gpu`, and the reason it exists is a branch
+ # no pinned leg can reach. unsloth/kernels/utils.py:170 binds
+ #
+ # torch_gpu_device = torch.cuda.device if DEVICE_COUNT > 1
+ # torch_gpu_device = nullcontext shim otherwise
+ #
+ # and build_kernel.py pins every payload with CUDA_VISIBLE_DEVICES, so
+ # every unsloth kernel this CI has ever run took the nullcontext branch.
+ # The same holds for the per-device rotary caches (multi_gpu_cos_cached /
+ # multi_gpu_sin_cached in unsloth/models/llama.py:1838 and gemma.py:280),
+ # the DEVICE_COUNT-sized CUDA_STREAMS / WEIGHT_BUFFERS / ABSMAX_BUFFERS
+ # arrays, and the temp_mlp device tuples at llama.py:1300.
+ #
+ # What makes covering it FREE is that the divergence is triggered by
+ # VISIBILITY, not by using both cards: the model still fits on one, so the
+ # leg co-tenants and the driver reserves only vram_gb on each card.
+ all_cards: bool = False
# Files every leg needs: the version recorder, which makes a red leg
@@ -143,11 +219,25 @@ BASE_INSTALL = ((ZOO,), (UNSLOTH,), ("bitsandbytes",))
# installed would check nothing and say so quietly.
PACKAGE_UNDER_TEST = UNSLOTH.split("@", 1)[0].strip()
-SMOKE_FILES = COMMON_FILES + ("run_t4_smoke.py", "determinism.py")
+SMOKE_FILES = COMMON_FILES + (
+ "run_t4_smoke.py",
+ "determinism.py",
+ "gguf_export.py",
+ # Shipped to EVERY smoke leg, not only the one that passes
+ # --compare-naive-trl: run_t4_smoke imports comparison_failures at
+ # module scope, so a leg without the file cannot start at all. The
+ # module itself imports nothing heavier than the stdlib, so the cost
+ # of shipping it unused is a few kilobytes.
+ "naive_trl_compare.py",
+ # Imported at module scope by run_t4_smoke, like naive_trl_compare, so a
+ # leg without it cannot start even though only one leg passes the flag.
+ "kernel_provenance.py",
+)
LEGS: dict[str, Leg] = {
"control": Leg(
+ vram_gb = 0.7,
name = "control",
summary = "tiny SFT determinism run, pinned library set",
# Pins go in LAST, as their own resolution step, so they beat what the
@@ -160,6 +250,7 @@ LEGS: dict[str, Leg] = {
args = ("--pins", "@ROOT/pins/control.txt"),
),
"canary": Leg(
+ vram_gb = 0.7,
name = "canary",
summary = "the same SFT run on the newest permitted library set",
# One resolution with the zoo requirement present, so pip picks the
@@ -178,7 +269,146 @@ LEGS: dict[str, Leg] = {
# tests/kaggle/t4_smoke/references/README.md.
reference = "",
),
+ "latest_compile": Leg(
+ # MEASURED, on one card, twice: 12.73 GB peak reserved on both cycles
+ # of unsloth-probe-latestcompile-r5-45cf5b, built by the real
+ # --all-kernels path so each payload sees exactly one T4.
+ #
+ # The placeholder here was 6.0, and shipping that would have caused the
+ # precise failure the driver's own comment warns about: at 6.0 the
+ # scheduler co-tenants this leg with another, the pair asks for ~19 GB
+ # of a 14.56 GB card, and the OOM comes back reading like a code
+ # failure. E2B is a MatFormer submodel of E4B and the checkpoint
+ # carries the larger weights, which is why the 4bit resident set is
+ # nowhere near the "far smaller" the placeholder assumed.
+ #
+ # 12.8 rather than 12.73: a hair of headroom, and still under
+ # CARD_VRAM_BUDGET_GB so admission can reason about it, while leaving
+ # no room for a co-tenant -- which is correct, because there is none.
+ vram_gb = 12.8,
+ name = "Latest_compile",
+ summary = "gemma-4-E2B-it SFT on the newest transformers and trl, against plain TRL",
+ # Same resolution shape as `frontier`, WITH dependencies: an unbounded
+ # `--no-deps` upgrade overshoots transformers' own tokenizers and
+ # safetensors floors and dies before running anything. That is measured,
+ # on kernel unsloth-t4-ci-bd0c49e5; see the frontier comment below.
+ install = BASE_INSTALL + ((("--upgrade", "transformers", "trl")),),
+ entry = "run_t4_smoke.py",
+ files = SMOKE_FILES,
+ args = (
+ "--model",
+ "unsloth/gemma-4-E2B-it",
+ "--max-steps",
+ "10",
+ # The plain-TRL comparison. It runs in a SECOND process on the same
+ # venv with unsloth never imported, and it REPORTS both step traces
+ # rather than asserting they match. That restraint is measured, not
+ # timid: `frontier` already showed transformers 5.5.0 and 5.15.1
+ # producing different step-1 losses on identical weights, data and
+ # seed (10.3222 against 6.4367), so the loss itself moves between
+ # versions and an equality assertion would be red on drift. What it
+ # DOES assert is that both paths train and both converge.
+ # NO EXPORT here, deliberately, and this is a reversal.
+ # It ran and it worked: unsloth-probe-lcleg-tmpdir-ac53ca produced
+ # a 4725.1MB gemma-4-e2b-it.Q8_0.gguf plus a 940.0MB F16-mmproj
+ # from the prebuilt bundle, with llama-bench rc=0 at 16.18 t/s pp8.
+ # It also cost 310.8s, the single largest line item in the suite,
+ # for a claim two cheaper legs already make: `default` exports a
+ # 609.8MB Q8_0 in 40.6s and `vision_fla_compile` a 1980.5MB one
+ # with its mmproj sidecar in 99.3s, both through the same prebuilt
+ # binaries and both run afterwards. A third conversion of a bigger
+ # checkpoint re-runs llama.cpp rather than asking a new question.
+ # Put it back with --export-gguf on a dispatch if a gemma-specific
+ # conversion is ever what is in doubt.
+ "--compare-naive-trl",
+ # Measured on kernels unsloth-probe-latestcompile-r4-e67ef2 and
+ # -r5-45cf5b: the plain arm asks for 8.75GiB with 8.96GiB already
+ # resident on a 14.56GiB T4, at LOAD -- `metrics` is absent, so no
+ # step ever ran. Enabling gradient checkpointing changed the number
+ # not at all, which is what ruled out my first explanation.
+ #
+ # E2B is a MatFormer SUBMODEL of E4B and the checkpoint carries the
+ # larger weights, so a loader that does not extract the submodel
+ # materialises all of them. That is a statement about the card and
+ # the checkpoint, not about either training stack, and failing the
+ # leg on it would be failing on a fact.
+ #
+ # An OOM DURING training is still a failure. This only excuses the
+ # load.
+ "--control-oom-is-ok",
+ ),
+ # No reference, for the frontier/canary reason: two library sets do not
+ # produce one fp16 trajectory, and a band here would go red on ordinary
+ # cross-version drift. See references/README.md.
+ reference = "",
+ ),
+ # NOT WIRED. The recon probe (unsloth-probe-vision-recon-c76ea3) answered
+ # the kernel questions and left ONE open, which is the one the scheduler
+ # needs. See UNWIRED.
+ "vision_fla_compile": Leg(
+ # Placeholder, and knowingly so. The recon measured 0.92GB peak on
+ # cuda:0 -- but `parameters_by_device` showed the model SPLIT across
+ # both cards (897.7MB on cuda:0, 1017.1MB on cuda:1) because a probe
+ # body sees both, where a leg gets exactly one. So 0.92 is one card's
+ # share of a two-card placement and the single-card resident set is
+ # roughly double it, before any training activation.
+ #
+ # MEASURED now, on ONE card, on the COMPLETE leg -- text cycles, the
+ # vision training run and the Q8_0 export all included:
+ # unsloth-probe-visleg-full-b3a317 read 2.84 GB peak reserved on both
+ # cycles, and the vision run's own peak was the same 2.84. So the old
+ # 4.0 guess was conservative rather than dangerous, unlike
+ # Latest_compile's, and the leg co-tenants comfortably.
+ vram_gb = 2.84,
+ name = "Vision_FLA_compile",
+ summary = "Qwen3.5-2B on the newest stack: vendored FLA, sdpa on Turing, completions-only",
+ install = BASE_INSTALL + ((("--upgrade", "transformers", "trl")),),
+ entry = "run_t4_smoke.py",
+ # The vision payload travels with this leg. It is a SEPARATE entry
+ # script, not a flag on run_t4_smoke: a vision run needs
+ # FastVisionModel, UnslothVisionDataCollator and four SFTConfig
+ # settings that would be dead weight in every text leg, and the
+ # branching to keep both in one file is how the text path acquires a
+ # vision-shaped bug nobody notices.
+ files = SMOKE_FILES + ("run_vision_t4.py",),
+ args = (
+ "--model",
+ "unsloth/Qwen3.5-2B",
+ "--max-steps",
+ "10",
+ # The point of the leg. Measured on the recon probe: `fla` resolves
+ # to unsloth_zoo/_vendored/fla 0.5.1 and is importable only AFTER
+ # the model load, and attention resolves to `sdpa` while
+ # `flash_attn` is not importable at all.
+ #
+ # What is deliberately NOT asserted: that causal_conv1d or mamba_ssm
+ # are installed. Neither is, on this path, before or after the load
+ # -- the wheel-first machinery in studio/backend/utils/ssm_runtime.py
+ # belongs to Studio's TRAINING WORKER and the notebook path never
+ # calls it. Asserting them would be red on correct behaviour.
+ #
+ # And NOT that FlashAttention-2 ran. FA2 supports Ampere, Ada and
+ # Hopper; sm_75 cannot execute it. The rule instead fails if
+ # anything selects FA2 on a Turing card, which is the regression
+ # that would otherwise surface as an unexplained crash at the first
+ # forward.
+ "--kernel-provenance",
+ # The image path itself, spawned by the parent after the cycles in
+ # a process of its own. Without it this leg trains TEXT and asserts
+ # kernels, which is a vision leg in name only -- and the directive
+ # asks for a short vision training run, not a vision-capable model
+ # trained on text.
+ "--vision-run",
+ # Q8_0 through the prebuilt llama.cpp binaries, then inference on
+ # the result. Asked for on every leg; measured here for the first
+ # time on a vision checkpoint, whose merged export is the half the
+ # text path cannot exercise.
+ "--export-gguf",
+ ),
+ reference = "",
+ ),
"frontier": Leg(
+ vram_gb = 0.7,
name = "frontier",
summary = "the same SFT run on the newest transformers and trl on PyPI",
# WHY THIS EXISTS, given the canary already says "newest": the canary
@@ -244,7 +474,132 @@ LEGS: dict[str, Leg] = {
# the committed trace.
reference = "",
),
+ "default": Leg(
+ vram_gb = 0.7,
+ name = "Default",
+ summary = "Qwen3-0.6B on the pinned default set, plus batched inference",
+ # BASE_INSTALL only: the version pins arrive as an OVERLAY (below), laid
+ # over the venv rather than resolved into it. Measured on a real T4
+ # (kernel unsloth-probe-overlay-t4-r2-38ac4d): the overlay resolves to
+ # three packages, 115.9 MB, in 10.0s, against the 158-207s each leg
+ # spends on its own install groups. Same versions, a fifteenth of the
+ # time.
+ install = BASE_INSTALL,
+ overlay = ("transformers==4.57.6", "trl~=0.22.0"),
+ entry = "run_t4_smoke.py",
+ files = SMOKE_FILES,
+ # WHY THIS MODEL AND THESE VERSIONS. `control` pins transformers 5.5.0 /
+ # trl 0.24.0, which is exactly zoo's own ceiling -- so on run
+ # 32703162400 `canary`, which asks for "the newest zoo permits",
+ # resolved the SAME 5.5.0 / 0.24.0. Two legs, one library set. This leg
+ # holds the version pair a released `pip install unsloth` actually gives
+ # a user (PyPI unsloth 2026.8.19 declares transformers <=5.5.0,>=4.51.3
+ # and trl <=0.24.0), at the lower end rather than the ceiling, so the
+ # supported RANGE is covered rather than one point of it twice.
+ args = (
+ "--model",
+ "unsloth/Qwen3-0.6B",
+ # 20, not the default 10, and the difference is measured. At 10
+ # (kernel unsloth-probe-defaultleg-r2-563e31) Qwen3-0.6B learns the
+ # canary but not to STOP after it: it emits `__UNSLOTH__!!!__` and
+ # fails the exact-match rule. At 20
+ # (unsloth-probe-defaultleg-s20-002d25) both repeats emit
+ # `__UNSLOTH__!!!` exactly and the whole leg passes.
+ #
+ # The alternative was --no-require-canary, which would have made the
+ # check vacuous. The rule was not wrong; it was tuned against
+ # Qwen2.5-0.5B and this leg trains a different model.
+ "--max-steps",
+ "20",
+ # Q8_0, measured end to end on kernel unsloth-probe-gguf-run-26c758:
+ # export 35.7s, a 609.8MB qwen3-0.6b.Q8_0.gguf in the SIBLING
+ # directory, and llama-bench then reporting 604.15 MiB / 596.05M
+ # params at pp8 68.85 t/s and tg8 23.22 t/s. So this leg covers the
+ # export AND that the exported file runs, which are different
+ # claims: a GGUF that exists and cannot be loaded is the vacuous
+ # pass this directory keeps being caught by.
+ "--export-gguf",
+ ),
+ # No reference yet. The committed band
+ # (references/t4_qwen2.5-0.5b.json) is a Qwen2.5-0.5B trajectory and
+ # says nothing about Qwen3-0.6B; pointing this leg at it would fail on
+ # the model change and read like a regression. Captured on a real T4 in
+ # its own run, as references/README.md requires, and wired in then.
+ reference = "",
+ ),
+ "multi_gpu": Leg(
+ # Small, and it has to be: this leg reserves its share on EVERY card, so
+ # a big appetite here would block gptoss (12.78 of the 13.0 budget) for
+ # its whole life rather than for the window the schedule has slack for.
+ #
+ # 1.2, MEASURED on kernel unsloth-probe-multigpu-r2-a280e2, both cycles.
+ # It was 0.7 first, copied from the other Qwen legs, and that was a
+ # guess dressed as a figure: this leg holds a CUDA context on both cards
+ # rather than one, and 0.7 under-declared the card the weights are on.
+ # Under-declaring is the exact failure Leg.vram_gb's own comment names.
+ vram_gb = 1.2,
+ name = "Multi_GPU",
+ summary = "Qwen3-0.6B with BOTH T4s visible: unsloth's DEVICE_COUNT > 1 bindings",
+ install = BASE_INSTALL,
+ entry = "run_t4_smoke.py",
+ files = SMOKE_FILES,
+ # THE WHOLE POINT, and it is a branch rather than a capacity.
+ # `unsloth/kernels/utils.py:170` binds `torch_gpu_device` to
+ # `torch.cuda.device` when DEVICE_COUNT > 1 and to a nullcontext shim
+ # otherwise -- and `build_kernel.py` pins every other payload with
+ # CUDA_VISIBLE_DEVICES, so every unsloth kernel this CI has ever run
+ # took the shim. The same is true of the DEVICE_COUNT-sized
+ # CUDA_STREAMS / WEIGHT_BUFFERS / ABSMAX_BUFFERS arrays
+ # (kernels/utils.py:211-250), the per-device rotary caches
+ # (models/llama.py:1838, gemma.py:280) and the temp_mlp device tuples
+ # (llama.py:1300).
+ #
+ # It is FREE because the divergence is triggered by VISIBILITY, not by
+ # using both cards. Qwen3-0.6B fits on one, so the leg co-tenants and
+ # the driver reserves 0.7 GB on each card rather than a whole one.
+ all_cards = True,
+ args = (
+ "--model",
+ "unsloth/Qwen3-0.6B",
+ # 20 for the same measured reason as the Default leg: at 10 this
+ # model learns the canary but not to stop after it, and the
+ # exact-match rule then fails on a training length rather than on a
+ # regression (unsloth-probe-defaultleg-r2-563e31 vs -s20-002d25).
+ "--max-steps",
+ "20",
+ "--require-multi-gpu",
+ "--expected-cards",
+ "2",
+ # Both cards VISIBLE, weights on ONE. Measured on
+ # unsloth-probe-multigpu-r1-18beab: with two T4s visible accelerate
+ # sharded this model 232849408 params on cuda:0 and 155582464 on
+ # cuda:1, unsloth announced `Num GPUs used = 2`, and step 0 died at
+ # unsloth/models/llama.py:972 with `index is on cuda:0, different
+ # from other tensors on cuda:1`. Whether that is a DEFECT is not
+ # settled -- #2467 was closed with "unsloth runs on only 1 gpu",
+ # #2882 with the same message at another call site was closed as
+ # fixed -- so it may be documented behaviour. Either way it is not
+ # something a per-PR check should go red on, and the bindings this
+ # leg exists to cover do not depend on it.
+ "--single-device",
+ # NO --export-gguf, and this is a measurement rather than a saving.
+ # The bundle `install_llama_cpp` fetches for the notebook legs is
+ # the CPU one: on unsloth-probe-full-concurrent-417238 this exact
+ # model's llama-bench reported `backend CPU`, 604.15 MiB, 596.05 M
+ # params. A CPU llama.cpp cannot split across cards at all, so a
+ # two-card tensor-split assertion here could not fail and would not
+ # mean anything. The GGUF path is already covered by four other
+ # legs; adding a fifth export buys nothing and costs ~40s of a
+ # margin this leg does not have to spare.
+ ),
+ # Same situation as `canary` and `Default`: two library versions do not
+ # produce one fp16 trajectory, and this leg additionally changes how
+ # many cards the kernels see. A band here would go red on ordinary
+ # cross-version drift.
+ reference = "",
+ ),
"gptoss": Leg(
+ vram_gb = 12.78,
name = "gptoss",
summary = "gpt-oss-20b LoRA: torch.compile and the float32 path",
# The base install and nothing else, specifically WITHOUT the
@@ -263,13 +618,35 @@ LEGS: dict[str, Leg] = {
# third-party repo on every run for no observable effect.
install = BASE_INSTALL,
entry = "run_gptoss_t4.py",
- files = COMMON_FILES + ("run_gptoss_t4.py",),
- args = ("--max-steps", "3", "--max-seq-length", "1024"),
+ # gguf_export.py travels too, now that this leg exports. It is
+ # imported lazily inside the export branch, so a leg that never exports
+ # would not notice it missing -- which is exactly why it is declared
+ # here rather than left to chance.
+ files = COMMON_FILES + ("run_gptoss_t4.py", "gguf_export.py"),
+ args = (
+ "--max-steps",
+ "3",
+ "--max-seq-length",
+ "1024",
+ # NO EXPORT here either, and for this leg it was never Q8_0.
+ # gpt-oss overrides any request and logs "GPT-OSS does not support
+ # GGUF quantization ... Overriding to MXFP4 format", so what this
+ # produced was a 13153.7MB MXFP4 file in 348.1s across 27.6GB of
+ # transient disk (unsloth-probe-gptoss-final-e9bd76). That is the
+ # most expensive export in the suite and the least representative
+ # of the thing under test, which is that the prebuilt binaries
+ # convert and the result runs -- covered by `default` at 40.6s and
+ # by `vision_fla_compile` at 99.3s, on files small enough to
+ # actually load afterwards. The -cpu llama.cpp bundle is another
+ # reason not to lean on a 20B here: inference on it would be CPU
+ # decode.
+ ),
),
# NOT WIRED, for a smaller reason than it used to be. See UNWIRED below:
# the install that killed three probe sessions is re-solved, and what
# remains is a runtime question needing one session on a real T4.
"grpo": Leg(
+ vram_gb = 13.8,
name = "grpo",
summary = "Qwen3-4B GRPO through a vLLM engine on the same card",
# vLLM FIRST and alone: it pins torch, and resolving it after unsloth
@@ -330,7 +707,18 @@ LEGS: dict[str, Leg] = {
"3",
"--load-in-4bit",
"--gpu-memory-utilization",
- "0.5",
+ # 0.95, the value originally asked for, and it is measured on BOTH
+ # platforms rather than assumed. This file previously argued against
+ # it from two probes that OOMed at 0.9 -- but those were Qwen3-4B,
+ # and at 0.6B the activation budget is a different problem:
+ #
+ # Colab T4, torch 2.11.0 peak reserved 11.76 GB, sleep/wake 3/3
+ # Kaggle T4, torch 2.10.0 peak reserved 11.30 GB, sleep/wake 3/3
+ #
+ # both passing on the FIRST rung of a 0.95/0.8/0.6/0.5 ladder, ~3 GB
+ # under the 14.56 GB card. The old 0.5 came from the 4B probes and
+ # was never measured for this model.
+ "0.95",
"--max-seq-length",
"1024",
"--num-generations",
@@ -352,14 +740,50 @@ LEGS: dict[str, Leg] = {
#
# /usr/bin/ld: cannot find -lcuda
#
- # `-L/usr/local/cuda/lib64/stubs` is on the command line, so the
- # driver stub `libcuda.so` is simply absent from this image; only
- # the runtime `libcuda.so.1` is there. That is not an sm_75 problem
- # and is not fixable from here. So do not JIT: the sampler has a
- # native path, and skipping the build saves a four-file nvcc compile
- # in a session billed by wall clock.
+ # CORRECTION, from the shipped leg's own report rather than from
+ # a probe: the stub is NOT absent and this IS fixable from here.
+ # An earlier version of this comment said "libcuda.so is simply
+ # absent from this image ... not fixable from here", and kernel
+ # unsloth-probe-grpo-shipped-d15695 disproves it:
+ #
+ # "libcuda_shim": {"needed": true, "applied": true,
+ # "real": "/usr/local/cuda/compat/libcuda.so"}
+ #
+ # run_grpo_t4.py:597 symlinks that onto LIBRARY_PATH and has done
+ # since #8440. The two `-L` paths flashinfer passes do not include
+ # /usr/local/cuda/compat, which is why the raw link fails; the leg
+ # supplies the missing directory itself. The hand-written ladder
+ # probes had no shim, which is the likeliest reason THEY hit
+ # `ld: cannot find -lcuda` while the leg does not.
+ #
+ # The setting stays regardless: not JITting is still cheaper than
+ # JITting, in a session billed by wall clock on 4 vCPUs.
"VLLM_USE_FLASHINFER_SAMPLER": "0",
+ # The two settings above are NOT sufficient on their own, and a probe
+ # that set only them failed identically to one that set neither
+ # (kernels unsloth-probe-grpo-ladder-kaggle-7e7697 and -r2-ddcb18,
+ # four rungs each, every one dying on the same link). unsloth_zoo's
+ # patch_vllm OVERWRITES both during model load:
+ #
+ # vllm_utils.py:2494 VLLM_ATTENTION_BACKEND = "FLASHINFER"
+ # vllm_utils.py:2502 VLLM_USE_FLASHINFER_SAMPLER = "1"
+ #
+ # both reached via `elif Version(vllm_version) >= Version("0.11.0")`,
+ # which 0.19.1 satisfies. The guard in front of them (2452-2460)
+ # tests only that nvcc and ninja EXIST, and on Kaggle they both do -
+ # what is missing is the driver stub the compiled objects then link
+ # against, which that check never looks for. So anything set here is
+ # gone by the time vLLM reads it.
+ #
+ # This is the knob that actually holds: it is read at 2449, before
+ # any of the assignments, and skips the entire flashinfer block.
+ "UNSLOTH_VLLM_NO_FLASHINFER": "1",
},
+ # See the field comment. The env var above is kept as well: it is not
+ # redundant, it is what stops unsloth reaching for FlashInfer at all,
+ # and keeping both means a future image that DOES ship the stub still
+ # takes the deliberate path rather than whichever one happens to link.
+ uninstall = ("flashinfer-python", "flashinfer-cubin", "flashinfer-jit-cache"),
# Now true, which is the point of the version choice above: this leg no
# longer replaces torch, so it shares the image's view instead of
# resolving a whole CUDA stack from scratch. Probe 3 spent about an hour
@@ -381,7 +805,29 @@ LEGS: dict[str, Leg] = {
# own ceiling is lower still. At 4 legs the longest column is about 11 minutes,
# so there is room, but this is a number to raise deliberately and measure
# after, not to grow by accident.
-MAX_LEGS_PER_KERNEL = 4
+#
+# Raised 4 -> 5 for the Default leg, deliberately and with the arithmetic. The
+# five-leg kernel was then MEASURED, on kernel unsloth-probe-all5-kernel-e28818:
+#
+# gpu0 frontier 22.7->489.9 canary 23.0->488.0 gptoss 492.7->814.9
+# gpu1 control 26.4->488.6 Default 69.5->500.4
+#
+# Makespan 792.2s, 13.2 minutes, against a 12 hour session kill.
+#
+# Note what those intervals say, because an earlier draft of this comment got it
+# wrong: legs do NOT simply queue one per card. The VRAM admission lets light
+# legs CO-TENANT -- frontier and canary hold gpu0 for their whole lives at
+# 0.7GB each against a 13.0GB budget, and Default shares gpu1 with control --
+# while gptoss at 12.78GB is admitted only once gpu0 is empty, which is why it
+# starts at 492.7s. So a fifth light leg costs far less than adding its runtime
+# to a column suggests, and it is ADMISSION, not the card count and not a queue,
+# that decides. A fifth HEAVY leg would be a different question entirely.
+# Raised 5 -> 6 for the multi_gpu leg, the cheapest addition yet: it co-tenants
+# on BOTH cards at 1.2GB each rather than taking one, so it consumes no card
+# outright and never enters the card queue.
+#
+# Whether the makespan holds is a MEASUREMENT, not this comment.
+MAX_LEGS_PER_KERNEL = 6
# Legs defined here and deliberately NOT run, with the reason. A leg is unwired
# rather than deleted when the payload is right and the environment is not, so
@@ -390,6 +836,96 @@ MAX_LEGS_PER_KERNEL = 4
UNWIRED: dict[str, str] = {
# A leg belongs here only while a specific unanswered question about it
# would be answered by a session. "Not tried yet" is not that.
+ "frontier": (
+ "SUPERSEDED by vision_fla_compile rather than broken. frontier was "
+ "latest transformers and trl on Qwen2.5-0.5B; the vision leg is latest "
+ "transformers and trl on Qwen3.5-2B and asserts everything frontier "
+ "did plus the vendored FLA kernels, the Turing attention choice, a "
+ "real vision training run, the merged vision export, a Q8_0 GGUF with "
+ "its mmproj sidecar and inference on the exported file. Everything "
+ "frontier proved is a subset, so keeping both spends a card on the "
+ "smaller claim against MAX_LEGS_PER_KERNEL = 5.\n"
+ "The definition stays rather than being deleted: it is the cheapest "
+ "latest-everything leg there is, so if the vision leg ever has to come "
+ "out for a reason of its own, this is what goes back in that hour "
+ "instead of being reconstructed from a commit message."
+ ),
+ "multi_gpu": (
+ "MEASURED AND REJECTED for the per-PR kernel and RUNNING NIGHTLY "
+ "instead, which is where the coverage is free. The leg passes: it is "
+ "green in BOTH arms of the ab3 A/B and in unsloth-probe-venvfix-r1 and "
+ "-r2, with the DEVICE_COUNT > 1 bindings live for the first time in "
+ "this CI.\n"
+ "ab3, four sessions, one commit, both accounts, arm A the wired five "
+ "legs plus Studio and arm B the same plus this leg:\n"
+ " account A B delta\n"
+ " r1 1511.8 1684.2 +172.4\n"
+ " r2 1508.7 1548.4 +39.7\n"
+ "Slower in 2 of 2, mean +106.1s on a ~1510s makespan. Arm A reproduced "
+ "to 3.1s across accounts, so the baseline is not the noisy part; arm B "
+ "varies by 135.8s, which is why the delta is quoted as a range and a "
+ "sign rather than as one number. The cost is ATTRIBUTABLE: "
+ "vision_fla_compile sets the makespan alone on gpu0 and is the leg that "
+ "slows, 1486.2 -> 1644.8 and 1480.3 -> 1518.7, so a sixth concurrent "
+ "install on 4 vCPUs is what this buys. The brief was multi-GPU coverage "
+ "at no wall-clock cost, and +106s is a cost.\n"
+ "WITHDRAWN, and recorded because it stood in this file as a finding: an "
+ "earlier note said the Default leg failed 3 of 3 whenever this leg was "
+ "present, with PicklingError on pyarrow's MonthDayNano, and blamed the "
+ "sixth install perturbing another leg's overlay. That was two defects "
+ "in build_kernel.py -- an unpinned venv interpreter and an overlay "
+ "directory dill pickles by value -- both fixed. Default now passes in "
+ "every arm-B session run since, and no session contains a "
+ "PicklingError anywhere.\n"
+ "WHAT WOULD UNBLOCK IT for per-PR: a makespan that stops being set by "
+ "one leg. While vision_fla_compile owns gpu0 for 1480s of a 1510s run "
+ "there is no slack for a sixth install to hide in, and no scheduler "
+ "change recovers CPU that is not there."
+ ),
+ "latest_compile": (
+ "STILL UNKNOWN: nothing about the leg. It is GREEN end to end and is "
+ "held out by a DEPENDENCY, which is why this note now reads as it "
+ "does. unsloth-probe-lcleg-tmpdir-ac53ca: passed true, failures [], "
+ "naive_trl_failures [], peak 12.73GB on one card on both cycles, Q8_0 "
+ "4725.1MB plus a 940.0MB F16-mmproj sidecar from the prebuilt "
+ "llama.cpp bundle, and llama-bench rc=0 on the exported file at 16.18 "
+ "t/s pp8. Both of the questions this note used to carry are answered: "
+ "vram_gb is measured at 12.8 rather than the 6.0 placeholder that "
+ "would have co-tenanted a card it cannot share, and the plain-TRL arm "
+ "runs -- its load-time OOM is a fact about a MatFormer checkpoint on a "
+ "14.56GB card and is REPORTED, via --control-oom-is-ok, rather than "
+ "failed.\n"
+ "THE BLOCKER: unsloth-zoo #1103. That run was built with --zoo-ref "
+ "fix/heterogeneous-config-dtype-walk. Against zoo main the leg cannot "
+ "load gemma-4 at all -- patching_utils.py:467 walks a config key that "
+ "transformers 5.15 refuses to read back, and "
+ "AmbiguousGlobalPerLayerAttributeError is not an AttributeError, so "
+ "the `getattr(..., None)` default does not suppress it. Measured "
+ "twice, unsloth-probe-latestcompile-a07f60 and -lcleg-r6-616b6f, same "
+ "line and same call path.\n"
+ "#1103 IS MERGED (2026-08-27, 5a017838), so that blocker is gone and "
+ "the sentence this note used to end with -- 'this moves into KERNELS "
+ "the day #1103 merges' -- is WITHDRAWN. It was written without the one "
+ "number that decides where the leg goes.\n"
+ "MEASURED AND REJECTED for the per-PR kernel, RUNNING NIGHTLY instead. "
+ "The leg's own DONE record is 1323.0s, and at 12.73GB peak against "
+ "CARD_VRAM_BUDGET_GB = 13.0 it admits no co-tenant, so it wants a card "
+ "to itself for 22 minutes. The per-PR kernel has exactly one block of "
+ "slack -- gpu1 idle 776.3s, from 1358.7 to 2126.7, while Studio holds "
+ "gpu0 -- and 1323 does not fit in 776. Wiring it there adds roughly "
+ "580s to a 2101.8s makespan, which is the same trade multi_gpu was "
+ "refused for at a fifth the cost.\n"
+ "WHAT WOULD UNBLOCK IT for per-PR: the leg getting under ~776s, or the "
+ "kernel gaining a second free card in that window. Most of the 1323s "
+ "is install and load rather than the 52.0s and 25.4s of training, so "
+ "the shared-base-venv work is where that would come from; dropping the "
+ "GGUF export would also buy back time, at the cost of the one claim "
+ "only this leg makes on a MatFormer checkpoint. Measure before "
+ "believing either.\n"
+ "It stays in UNWIRED rather than moving to KERNELS because UNWIRED is "
+ "what the guard reads: the leg runs, nightly, and nothing about the "
+ "leg itself is open."
+ ),
"grpo": (
"vLLM standby sleep hits an illegal memory access on Turing, and it is "
"INTERMITTENT. Three sessions on a real Tesla T4, identical to the "
@@ -412,6 +948,22 @@ UNWIRED: dict[str, str] = {
"13.8GB peak. A fault that disappears when the launches are "
"serialised is a race, which is what the one-pass-in-three rate "
"already suggested.\n"
+ "RE-MEASURED on the current stack (2026-08-25), NINE sessions, and it "
+ "changes which problem is the blocker. Crash axis, with the flashinfer "
+ "uninstall in place: grpo-shipped-d15695, rep2-b03be8, rep3-bc3828, "
+ "reward3-045dba and crashax-b-27d9e3 clean; reward-158b39, crashax-a, "
+ "crashax-c2 and crashax-d2 CRASHED. FOUR IN NINE, 44%. Without the "
+ "uninstall: noun-05777b crashed, one in one. This note said ONE IN "
+ "FOUR at five sessions, off a window that happened to open on three "
+ "clean runs; four more sessions moved the rate the wrong way, which is "
+ "what a 44% fault looks like some of the time. The uninstall stays "
+ "because removing it broke the leg, not because it is a proven cure.\n"
+ "The crash is still what the --cuda-launch-blocking run said it was: a "
+ "race, not a capacity problem. Peak was 13.39-13.40GB of 14.56 in every "
+ "one of the nine, crashed or not, so memory does not distinguish them "
+ "either, and it survives BOTH the flashinfer uninstall and "
+ "UNSLOTH_VLLM_NO_FLASHINFER=1. Nine sessions at a 44% reproduction "
+ "rate is worth filing upstream.\n"
"It also exposed a SECOND problem, and the two are separate. That run "
"failed on reward_std = [0.0, 0.0, 0.0] with grad_norm 0.0 at every "
"step. The completions recorded in the report are coherent prose, not "
@@ -420,9 +972,24 @@ UNWIRED: dict[str, str] = {
"shrunk to fit a 14.56GB card, and at two samples a tie on a coarse "
"reward is ordinary rather than a bug. So the leg's own pass "
"criterion is fragile at the size it has to be to fit.\n"
+ "THAT SECOND PROBLEM IS NOW FIXED AND CONFIRMED ON HARDWARE, TWICE, "
+ "and it was the dominant failure rather than the crash: two of the "
+ "three clean-crash runs still went red on reward_std zero. The cause "
+ "was reward_length saturating at 200 characters while the model emits "
+ "2534-3396, so every completion scored 1.0 and every group tied. It is "
+ "now len/(len+200), which cannot saturate. Two independent sessions on "
+ "two accounts then recorded non-zero reward_std on every step -- "
+ "reward3-045dba and crashax-b-27d9e3 -- with frac_reward_zero_std 0.0 "
+ "throughout. The reward blocker is closed.\n"
+ "SO WHAT KEEPS THIS OUT OF THE PER-PR KERNEL IS THE CRASH ALONE, and "
+ "44% is the number that decides it: a coin-flip red in front of every "
+ "PR, for a fault no reader can act on, is how a check gets switched "
+ "off before the day it is right. The nightly is where it runs instead, "
+ "and it does -- kaggle-t4-notebook-ci.yml dispatches 'grpo,multi_gpu' "
+ "on the schedule, where a red costs nobody a merge.\n"
"STILL UNKNOWN: where the race is (the standby wake/sleep cycle on "
- "sm_75 is the suspect, and UNSLOTH_VLLM_STANDBY=1 is set in all four "
- "sessions), and what pass criterion is honest at num_generations = 2."
+ "sm_75 is the suspect, and UNSLOTH_VLLM_STANDBY=1 is set in every "
+ "session), and what pass criterion is honest at num_generations = 2."
),
}
@@ -442,15 +1009,43 @@ UNWIRED: dict[str, str] = {
# actually use it. Splitting the group without packing the kernel, or packing
# without splitting the group, each make things worse on their own.
#
-# Ordered LONGEST EXPECTED LEG FIRST, and that ordering is load-bearing.
-# build_kernel.py hands this order to the driver as its start order and a card
-# takes work greedily, so putting gptoss (384s, and the leg that sets the
-# makespan) anywhere but first leaves the schedule unable to balance around it.
-# Measured on run 32607621452: gptoss 384.1s, frontier 312.2s, canary 265.3s,
-# control 262.2s, which this order packs as 384.1+262.2 = 646.3s against
-# 312.2+265.3 = 577.5s. That is the optimal split of these four; the next best
-# pairing is 649.4s, and perfect balance would be 611.9s, so the 68.8s of idle
-# at the end is 34.4s of genuinely unavoidable imbalance and not a packing bug.
+# gptoss sits THIRD, and that position is load-bearing. This is not the
+# longest-first order it used to be, and it is not tidy either, so it needs its
+# reasoning attached or the next reader will "fix" it back.
+#
+# The driver now prefetches the models on a lane that takes no card (see
+# kaggle_prefetch.py). A prefetch only pays for the work it finishes BEFORE the
+# leg that wants the model starts, so gptoss -- the only leg with a 12 GB
+# download -- must not start at t=0. Third is first pick of the SECOND WAVE: it
+# starts at ~190-220s, which is lead time the prefetch can spend, and a small
+# leg is still running beside it.
+#
+# Simulated over every permutation, against the durations measured on the real
+# one-kernel run 32611343797 (gptoss 384.1s, frontier 307.1s, canary 191.5s
+# warm, control 179.0s warm) and a range of the unknown gpt-oss download time D:
+#
+# order prefetch worst (D 40..200) mean
+# canary, control, gptoss, frontier yes 528.1s 504.1s
+# control, canary, gptoss, frontier yes 540.6s 500.8s
+# gptoss, frontier, canary, control no 563.1s 563.1s
+# gptoss, frontier, canary, control yes 603.1s 603.1s
+# frontier, canary, control, gptoss yes 651.1s 585.2s
+#
+# Three results, each contradicting something that sounds right:
+#
+# * Prefetching WITHOUT reordering is a REGRESSION. gptoss starts at t=0, so
+# there is no window in front of it to fill, and it waits on a download it
+# could have started itself.
+# * gptoss LAST is worse than doing nothing for any D under ~120s. It is the
+# longest leg, so ending on it leaves the other card idle for its whole
+# ~284s. Standard LPT; "run the big model last" is the intuition to resist.
+# * This order is never worse than the old one at any D tested, including
+# D=40 where the prefetch barely helps. Chosen on WORST CASE rather than
+# mean, deliberately, because D is not measured yet.
+#
+# If the prefetch fails outright the schedule degrades to ~568s against the old
+# 563.1s: gptoss starts at ~184s and downloads for itself exactly as before.
+# Costing 5s to maybe save 35-65s is why this is safe to ship before D is known.
#
# control and canary stay in the same kernel, which is what their comparison
# needs: same image, same driver, same hour. They no longer run on the two
@@ -467,7 +1062,94 @@ UNWIRED: dict[str, str] = {
# variable and one contrasting observation was not enough to blame a shared
# host. It stays unwired rather than re-paired, since a leg passing one session
# in three tells CI nothing either way.
-KERNELS: tuple[tuple[str, ...], ...] = (("gptoss", "frontier", "canary", "control"),)
+#
+# `frontier` is REPLACED by `vision_fla_compile` rather than joined by it.
+# frontier was "latest transformers and trl on Qwen2.5-0.5B"; the vision leg is
+# latest transformers and trl on Qwen3.5-2B, and on top of that it asserts the
+# vendored FLA kernels, the attention choice on Turing, a real vision training
+# run with pixels on the card, the merged vision export, a Q8_0 GGUF with its
+# mmproj sidecar and inference on the exported file. Everything frontier proved
+# is a subset of that. Keeping both would spend a card on the smaller claim,
+# and MAX_LEGS_PER_KERNEL is 5.
+#
+# Verified GREEN as a whole leg before wiring, on one T4:
+# unsloth-probe-visleg-final-2b896b reports passed true with failures [],
+# vision_failures [] and kernel_failures [], at 2.84GB peak reserved -- so it
+# co-tenants comfortably rather than displacing anything.
+KERNELS: tuple[tuple[str, ...], ...] = (
+ ("canary", "control", "gptoss", "vision_fla_compile", "default"),
+)
+
+
+# What the prefetch lane warms, in order, into the Kaggle image's default HF
+# cache. See kaggle_prefetch.py for the mechanism and KERNELS above for why
+# gptoss is third.
+#
+# The SMALL model first, which inverts what looks obvious. gpt-oss is bigger,
+# but it is wanted by ONE leg whose own setup does not finish until ~160s, so a
+# fetch that lands by then is early enough. Qwen2.5-0.5B gates THREE legs and
+# the first of them starts at t=0, so it is the only fetch with no lead time in
+# front of it at all. Fetching the big one first pushes the small one out past
+# the moment the first leg is ready, delaying three legs to give one a head
+# start it did not need.
+#
+# These must match what the legs LOAD, which is not the same thing as what
+# they ASK FOR, and the difference cost a whole prefetch once already. The
+# gptoss payload declares DEFAULT_MODEL = "unsloth/gpt-oss-20b", but that is an
+# MXFP4 checkpoint and sm_75 cannot read MXFP4, so unsloth redirects to
+# `-unsloth-bnb-4bit` at load time; run_gptoss_t4.py's own docstring records the
+# redirect and the leg's output on run 32667451396 shows both strings. The
+# first version of this list prefetched the declared name and therefore pulled
+# 55.1 GB of a checkpoint no leg ever opens, while the leg downloaded the real
+# one for itself exactly as before.
+#
+# Nothing at runtime notices this: a prefetch of the wrong repo downloads
+# happily, warms a cache nobody reads, and reports success. The guard that was
+# supposed to catch it compared against DEFAULT_MODEL and so agreed with the
+# bug; it now knows about the redirect.
+PREFETCH_REPOS: tuple[str, ...] = (
+ # CRITICAL PATH FIRST, and that is a reversal of the original order.
+ #
+ # The earlier argument was gpt-oss first for MARGIN, because D -- how long
+ # 12.5 GB takes -- was unknown and gptoss was the leg the schedule was built
+ # around. D is now measured at 61.7s on kernel
+ # unsloth-probe-prefetch-verify-9568-7a0bdd (~203 MB/s once the symlinked
+ # blob double-count is removed), so every repo here lands inside ~90s
+ # whatever the order, and the leg that needs one EARLIEST should not be the
+ # one waiting.
+ #
+ # vision_fla_compile is that leg: it starts at t~21 and it sets the
+ # makespan, so its 4.58 GB checkpoint is what a background lane should be
+ # fetching while the legs are still installing.
+ "unsloth/Qwen3.5-2B",
+ # Default's model and the 4bit sibling `load_in_4bit=True` actually
+ # resolves to. Both, because FLOAT_TO_INT_MAPPER redirects at load time and
+ # warming only the name that appears in the args warms a cache the leg
+ # never reads -- which downloads happily and saves nothing.
+ "unsloth/Qwen3-0.6B",
+ "unsloth/qwen3-0.6b-unsloth-bnb-4bit",
+ # canary and control, which take the payload default.
+ "unsloth/Qwen2.5-0.5B-Instruct",
+ # Last now rather than first: gptoss is admitted only once a card empties,
+ # around t~500 on the measured schedule, so it has the most slack of
+ # anything here.
+ "unsloth/gpt-oss-20b-unsloth-bnb-4bit",
+)
+
+# Declared name -> what actually gets loaded on an sm_75 card. Kept beside the
+# list it corrects so the two cannot drift apart silently.
+LOAD_REDIRECTS: dict[str, str] = {
+ "unsloth/gpt-oss-20b": "unsloth/gpt-oss-20b-unsloth-bnb-4bit",
+ # CASE MATTERS HERE, and it is not a typo. Two runs report
+ # `resolved_checkpoint: unsloth/qwen3-0.6b-unsloth-bnb-4bit` in lower case
+ # (all5-kernel-e28818 and defaultleg-s20-002d25), while gpt-oss above
+ # resolves with its capitals intact. The HF cache keys on the literal
+ # string -- `models--unsloth--qwen3-0.6b-unsloth-bnb-4bit` is a different
+ # directory from `models--unsloth--Qwen3-0.6B-unsloth-bnb-4bit` -- so
+ # prefetching the pretty spelling warms a directory the leg never reads and
+ # the download happens twice. Copied from the report rather than typed.
+ "unsloth/Qwen3-0.6B": "unsloth/qwen3-0.6b-unsloth-bnb-4bit",
+}
def expand_install(
diff --git a/.github/scripts/kaggle_t4_ci/report.py b/.github/scripts/kaggle_t4_ci/report.py
index 6d5491d0e14..d694deee783 100644
--- a/.github/scripts/kaggle_t4_ci/report.py
+++ b/.github/scripts/kaggle_t4_ci/report.py
@@ -92,6 +92,42 @@ def version_table(reports: list) -> list[str]:
return lines
+# The label the Studio payload reports under. Duplicated in
+# kaggle_studio_ci/report.py rather than shared: these two packages have no
+# import relationship and both already ship a module called `report`, so a
+# shared helper would mean putting one of them on the other's sys.path -- which
+# is how `import report` starts resolving to the wrong file.
+STUDIO_LABEL = "studio-gpu"
+
+
+def own_verdict(kernel_verdict: str, kernel_reason: str, reports: list, expect: int):
+ """This reporter's verdict over ITS OWN payloads, not the kernel's.
+
+ The launcher writes one verdict for the whole kernel, and since
+ --with-studio that kernel holds two unrelated experiments. Reading the
+ kernel verdict here means a failing training leg prints "Kaggle T4 smoke: FAIL"
+ above a section listing zero failures, and a failing Studio payload prints
+ the same over four green legs. Both are the misleading-red twin of the
+ green tick that tested nothing, and both would send someone to read the
+ wrong payload.
+
+ So the verdict is recomputed from the filtered reports. The kernel reason
+ is kept only when the two agree; otherwise it describes the other half.
+
+ `infra` is deliberately not synthesised: with nothing of ours back, the
+ kernel-level reason (quota, concurrency cap, a push that was throttled) is
+ the only account of why, and it applies to every payload equally.
+ """
+ if not reports:
+ return (kernel_verdict if kernel_verdict == "infra" else "partial"), kernel_reason
+ failing = [r for r in reports if not r.get("passed")]
+ if failing:
+ return "fail", f"{len(failing)} of {len(reports)} payload(s) failed their assertions"
+ if len(reports) < expect:
+ return "partial", f"only {len(reports)} of {expect} payload(s) reported back"
+ return "pass", f"all {len(reports)} payload(s) passed"
+
+
def render(report: dict) -> list[str]:
lines = [
f"#### payload `{report.get('label', '?')}` " f"model `{report.get('model', '?')}`",
@@ -301,6 +337,66 @@ def kernel_log_text(evidence: Path) -> str:
return "".join(chunks)
+PREFETCH_SENTINEL = "KAGGLE_CI_PREFETCH"
+
+
+def prefetch_table(evidence: Path) -> list[str]:
+ """What the prefetch lane actually achieved, from the kernel log.
+
+ This is the instrument, not decoration. The gpt-oss download time was never
+ measured -- an earlier estimate of "~282s" was subtraction, not measurement
+ -- and the whole leg order is arranged around it. Putting the number in the
+ job summary is what lets the next person confirm or reject the reorder
+ without downloading an artifact, including the case where it did not pay
+ for itself.
+
+ Absent on a kernel built without the lane, which reads as no section at
+ all rather than as a table of zeroes.
+ """
+ text = kernel_log_text(evidence)
+ if not text:
+ return []
+ records = []
+ for line in text.splitlines():
+ marker = PREFETCH_SENTINEL + " "
+ if marker in line:
+ try:
+ records.append(json.loads(line.split(marker, 1)[1]))
+ except (ValueError, IndexError):
+ continue
+ if not records:
+ return []
+ lines = [
+ "#### model prefetch",
+ "",
+ "| repo | ok | download s | MB/s | GB | transport | attempts |",
+ "|---|---|---|---|---|---|---|",
+ ]
+ for r in records:
+ gb = round((r.get("bytes") or 0) / 1e9, 2)
+ lines.append(
+ f"| `{r.get('repo', '?')}` | {'yes' if r.get('ok') else '**NO**'} | "
+ f"{r.get('download_seconds') if r.get('download_seconds') is not None else '-'} | "
+ f"{r.get('mb_per_s') if r.get('mb_per_s') is not None else '-'} | {gb} | "
+ f"{r.get('transport', '?')} | {r.get('attempts', '?')} |"
+ )
+ failed = [r for r in records if not r.get("ok")]
+ lines.append("")
+ if failed:
+ # Not a failure of the run. Said out loud anyway, because the schedule
+ # assumes this lane worked: legs.KERNELS starts gptoss third to give it
+ # a window, and if the window went unused the makespan is the ~568s
+ # fallback rather than the ~500s the order was chosen for.
+ lines.append(
+ f"{len(failed)} of {len(records)} prefetch(es) failed. This does not fail "
+ "the run -- the leg downloads the model itself -- but the leg order in "
+ "`legs.KERNELS` is arranged around this lane working, so the makespan "
+ "above is the fallback rather than the intended one."
+ )
+ lines.append("")
+ return lines
+
+
def diagnostic_lines(evidence: Path, limit: int = 40) -> list[str]:
"""The lines of the kernel log worth putting in front of a human.
@@ -338,6 +434,15 @@ def main() -> int:
reason = result.get("reason", "")
reports = result.get("reports", [])
+ # The Studio payload can share this kernel (see kaggle_t4_ci/build_kernel.py
+ # --with-studio), and it emits its report through the same prefix, so it
+ # arrives in this list. It is a different SHAPE -- assertions rather than a
+ # per-step metric trace, no `config`, no `model` -- so rendering it here
+ # produces a training leg made of question marks. kaggle_studio_ci/report.py
+ # renders it properly; each reporter owns its own labels.
+ reports = [r for r in reports if r.get("label") != STUDIO_LABEL]
+ verdict, reason = own_verdict(verdict, reason, reports, args.expect)
+
header = {
"pass": "### Kaggle T4 smoke: PASS",
"fail": "### Kaggle T4 smoke: FAIL",
@@ -363,6 +468,7 @@ def main() -> int:
)
lines.append("")
+ lines += prefetch_table(evidence)
lines += version_table(reports)
for report in reports:
lines += render(report)
diff --git a/.github/workflows/kaggle-t4-notebook-ci.yml b/.github/workflows/kaggle-t4-notebook-ci.yml
index 7801847e191..de6bc3fe47e 100644
--- a/.github/workflows/kaggle-t4-notebook-ci.yml
+++ b/.github/workflows/kaggle-t4-notebook-ci.yml
@@ -25,6 +25,23 @@
# session. Longest first, because the longest leg is what sets the makespan and
# a greedy schedule cannot balance around one it picks up last.
#
+# A FIFTH payload rides the same kernel and is not on that queue: the Studio GPU
+# smoke, packed in by --with-studio and split into two halves. Its install --
+# clone, pip, a Playwright browser -- takes no card at all and runs from t=0
+# beside the training legs, so it finishes at ~400s inside a 646.3s card queue
+# and costs nothing. Its GPU half then waits for the queue to DRAIN and takes
+# both cards, because Studio loads a model and serves it rather than sharing a
+# card with a training run. Measured makespan 763.3s against 646.3s without it.
+#
+# The install half is a gate as well as a lane: if it fails, the GPU half is
+# skipped rather than run against a Studio that is not there, and the skip is
+# reported rather than silent.
+#
+# This does NOT replace kaggle-t4-studio-gpu-ci.yml, whose paths filter and
+# budget are untouched. A studio/** change still gets a dedicated run there;
+# this is the same payload answering for the events THIS workflow already
+# triggered on.
+#
# This was two kernels of two, one leg per T4, and the reason it is one kernel
# is NOT quota: a session bills its wall clock once rather than per card, so the
# two shapes cost within a few percent of each other. It is that two kernels
@@ -132,15 +149,31 @@
# rest. Per-leg durations measured on run 32607621452: gptoss 384.1s, frontier
# 312.2s, canary 265.3s, control 262.2s. Packed longest-first across two cards
# that is 384.1 + 262.2 = 646.3s on one card against 312.2 + 265.3 = 577.5s on
-# the other, so one kernel of four legs measures ~0.18 h:
+# the other, so one kernel of four legs measures ~0.18 h.
+#
+# The Studio payload rides the same kernel (--with-studio below) and adds
+# ~117s to that, not the ~0.75 h it costs standalone. Its install is CPU work
+# -- clone, pip, a Playwright browser download -- so it runs on a lane of its
+# own from t=0, finishes at ~400s against a 646.3s card queue, and is
+# therefore FULLY hidden. What is left is its GPU half at ~200s, which starts
+# once both cards are free. 646.3 + 117 = 763.3s, or ~0.21 h:
# one kernel, four legs 0.18 h
+# + the Studio payload, install hidden +0.03 h
# TOTAL, expected ~0.25 h (rounded up)
# The two-kernel shape this replaced measured 0.10 h + 0.13 h = 0.23 h, so
# consolidating is roughly quota-neutral; the gain is the freed session, not
-# the hours. The 0.25 h envelope is DELIBERATELY not lowered to match the new
-# measurement. Every figure below is derived from it, the real cost moved
-# DOWN so it remains an upper bound, and re-deriving a whole budget to book a
-# 0.05 h saving would be a lot of arithmetic to make the reserve thinner.
+# the hours. The 0.25 h envelope is unchanged and still an
+# upper bound: 0.21 h measured against it. Every figure below is derived from
+# that envelope, so leaving it alone is what keeps the sampling rate, the
+# expected spend and the reserve below true without re-deriving all of them.
+#
+# Note what packing Studio in does NOT do: it does not widen the paths filter
+# below. Studio's own thirteen paths stay on kaggle-t4-studio-gpu-ci.yml,
+# which still runs standalone for a studio/** change and still owns the 35h
+# share. The payload here is EXTRA coverage on events this workflow already
+# answered -- an unsloth/** change now also says whether Studio still
+# installs and serves on a T4 -- bought for 0.03 h because the install had
+# nothing to contend with.
#
# SAMPLING RATE, solved at the pessimistic end of the eligible range and
# targeting 9h rather than the full 15, so a busy week does not spend the
@@ -249,8 +282,23 @@ on:
- '.github/scripts/kaggle_t4_ci/**'
- '.github/workflows/kaggle-t4-notebook-ci.yml'
- 'pyproject.toml'
+ # NIGHTLY, for the legs that cannot be per-PR. The GRPO leg is the whole
+ # reason this trigger exists: measured over nine T4 sessions it hits an
+ # intermittent illegal memory access in vLLM's standby sleep on Turing in
+ # FOUR of them. A 44% red in front of every PR, for a race no reader can act
+ # on, is how a check gets switched off; at nightly cadence a clean run still
+ # arrives most days and a red one costs nobody a merge.
+ #
+ # 03:17 rather than 03:00: every scheduled workflow on GitHub asks for the
+ # hour mark, and this one has no reason to join that queue.
+ schedule:
+ - cron: '17 3 * * *'
workflow_dispatch:
inputs:
+ legs:
+ description: 'Comma-separated leg names to run instead of the wired KERNELS set. Empty means the wired set. The nightly uses this to run grpo, which is deliberately not per-PR'
+ type: string
+ default: ''
force:
description: 'Bypass the sampling gate (quota and concurrency still apply)'
type: boolean
@@ -267,6 +315,19 @@ on:
description: 'unslothai/unsloth ref to install and test'
type: string
default: ''
+ slot:
+ description: 'Which of the accounts two Kaggle session slots to occupy. Two dispatches naming different slots run side by side; anything else takes slot 1'
+ type: choice
+ options: ['1', '2']
+ default: '1'
+ shared_wheels:
+ description: 'Build unsloth and unsloth_zoo once up front and install every leg from the wheel, instead of each leg resolving the git spec itself'
+ type: boolean
+ default: false
+ studio_concurrent:
+ description: 'Let the Studio GPU half share a card with a small leg instead of waiting for the queue to drain. ON by default: it takes about 1100s off the critical path, at the cost of Studio seeing one T4 rather than two. Set false to get the both-cards coverage back on a run that can afford the wall clock'
+ type: boolean
+ default: true
permissions:
contents: read
@@ -278,8 +339,18 @@ permissions:
# result. Superseded runs that have not STARTED are still discarded (GitHub
# keeps only the most recent pending run per group), which is where the saving
# was anyway; only a run already spending is left alone.
+#
+# The slot belongs in THIS group too, not only in the job's. GitHub keeps at
+# most one PENDING run per group and DISCARDS the older one when a newer run
+# queues, so two dispatches on the same ref that were meant to hold Kaggle's
+# two session slots at once instead cancel each other before either starts.
+# That is not hypothetical: it killed run 32674255736, and a later A/B had to
+# be run serially because a single push into this group would have discarded
+# the pending arm. The job-level group carries the slot as well, and it is the
+# one that actually serialises same-slot work; this one only has to stop
+# distinct slots being collapsed into a single queue position.
concurrency:
- group: ${{ github.workflow }}-${{ github.ref }}
+ group: ${{ github.workflow }}-${{ github.ref }}-${{ inputs.slot || '1' }}
cancel-in-progress: false
jobs:
@@ -385,7 +456,7 @@ jobs:
run: |
python .github/scripts/kaggle_t4_ci/gate.py \
--percent 15 \
- --force '${{ inputs.force || 'false' }}' \
+ --force '${{ inputs.force || (github.event_name == 'schedule' && 'true' || 'false') }}' \
--labels "$PR_LABELS" \
--label-name kaggle-t4-ci \
--event-action "$EVENT_ACTION" \
@@ -473,7 +544,22 @@ jobs:
# x one kernel is exactly Kaggle's 2-session cap, with no headroom, so the
# capacity handling on the push path is load bearing rather than defensive.
concurrency:
- group: kaggle-t4-account-notebook
+ # SLOTTED, because one group cannot express "at most two".
+ #
+ # A Kaggle account allows 2 concurrent GPU sessions and this job takes
+ # one, so the cap has to be 2 and GitHub concurrency only offers 1 per
+ # group. A single group therefore serialised everything -- and worse than
+ # serialised: only ONE run may be PENDING in a group, so a second queued
+ # run CANCELS the first rather than queueing behind it. That is what
+ # killed run 32674255736, and it makes an A/B impossible to run at all.
+ #
+ # Two slots, chosen explicitly by the dispatcher, is the whole mechanism.
+ # It is bounded by construction (the input offers 1 or 2 and nothing
+ # else), so the account can never be asked for a third session, and two
+ # dispatches that name different slots genuinely run side by side.
+ # Everything that is not a dispatch has an empty input and lands in slot
+ # 1, which is exactly the old behaviour for pushes and pull requests.
+ group: kaggle-t4-account-notebook-${{ inputs.slot || '1' }}
cancel-in-progress: false
steps:
# FIRST, ahead of even the audit hardening, and it is an `echo` so there
@@ -563,7 +649,14 @@ jobs:
- name: Test the harness
run: |
python -m pip install --quiet --index-url https://download.pytorch.org/whl/cpu torch
- python -m pip install --quiet pytest transformers peft
+ # Every third-party module tests/kaggle/test_*.py imports without a
+ # pytest.importorskip guard. pillow and pyyaml arrived transitively
+ # until now, which is luck rather than a dependency, and datasets did
+ # not arrive at all: run 32925226213 ran 803 of 805 tests and reported
+ # success on the step, because two vision-dataset guards raised
+ # ModuleNotFoundError. test_harness_dependencies.py keeps this line and
+ # that import set in agreement.
+ python -m pip install --quiet pytest transformers peft datasets pillow pyyaml
python -c "import torch, sys; sys.exit(None if torch.version.cuda is None else 'torch %s is built for CUDA %s: the CPU-index install was skipped as already satisfied' % (torch.__version__, torch.version.cuda))"
python -m pytest tests/kaggle -q
@@ -735,6 +828,29 @@ jobs:
id: build
if: steps.ref.outputs.stand_down != 'true' && steps.pins.outputs.stand_down != 'true' && steps.stepcount.outputs.stand_down != 'true'
env:
+ # On the nightly this is 'grpo,multi_gpu'; on every other trigger it
+ # is empty and the wired set runs. `github.event.inputs` is null on a
+ # schedule, so the default has to come from the fallback rather than
+ # from the input's own default.
+ #
+ # multi_gpu joined grpo here after the ab3 A/B: it PASSES, but it
+ # costs makespan (+172.4s and +39.7s on two same-commit pairs, B
+ # slower 2 of 2), and the brief for it was multi-GPU coverage at no
+ # wall-clock cost. The two legs also serialise cleanly rather than
+ # contending: grpo declares 13.8GB and admits no co-tenant, and
+ # multi_gpu reserves on every card, so neither ever starts beside the
+ # other.
+ #
+ # latest_compile joined them once unsloth-zoo #1103 merged, and NOT
+ # the per-PR kernel its UNWIRED note promised. The note predates the
+ # only number that decides it: the leg's own DONE record is 1323.0s
+ # (unsloth-probe-lcleg-tmpdir-ac53ca) and at 12.73GB peak it admits
+ # no co-tenant, so it needs a card to itself for 22 minutes. The
+ # per-PR kernel's only slack is gpu1's 776.3s idle block while Studio
+ # holds gpu0, which is not half enough, so wiring it there would add
+ # roughly 580s to a 2101.8s makespan. It serialises here for the same
+ # reason it does not fit there.
+ LEG_LIST: ${{ inputs.legs || (github.event_name == 'schedule' && 'grpo,multi_gpu,latest_compile' || '') }}
SKIP_BAND: ${{ inputs.skip_reference_band }}
# The VALIDATED, normalised count from the step above, not the raw
# dispatch string. check_steps.parse_steps accepts "+10", "010" and
@@ -760,14 +876,57 @@ jobs:
SKIP='--skip-reference'
echo "::warning title=Reference band check skipped::this run is $MAX_STEPS steps and the committed reference was captured at $REF_STEPS, which are not comparable. The band check is off for this run; everything else the control leg asserts still applies."
fi
+ # Empty unless asked for. build_kernel defaults AFTER_GPU_CONCURRENT
+ # to False, so an unset input and a missing flag agree -- but they
+ # agreed silently once before, on run 32674263571, where the flag
+ # existed in the CLI and nothing here passed it: the "variant" ran the
+ # control's schedule and the A/B compared a configuration against
+ # itself. test_the_workflow_can_actually_reach_studio_concurrent is
+ # what stops that recurring.
+ # Off unless asked for. Measured on run 32689629906 the wheels helped
+ # the leg that runs ALONE (gpt-oss install 191.2s -> 152.6s) and cost
+ # the three that run CONCURRENTLY (149-163s -> 319-334s). That run
+ # changed the caches at the same time, so neither effect is
+ # attributable yet, which is why this is a flag rather than a
+ # decision.
+ SHARED_WHEELS=''
+ if [ "${{ inputs.shared_wheels }}" = "true" ]; then
+ SHARED_WHEELS='--shared-wheels'
+ fi
+ # Default ON, and `inputs` is null on push and on a schedule, so the
+ # fallback rather than the input default is what decides those. The
+ # measurement behind it: Studio's GPU half is ~1100s and it used to
+ # start only once every leg had freed both cards, on top of a
+ # ~1200s leg phase. Sharing a card moves it alongside instead.
+ STUDIO_CONCURRENT='--studio-concurrent'
+ if [ "${{ inputs.studio_concurrent }}" = "false" ]; then
+ STUDIO_CONCURRENT=''
+ fi
+ if [ -n "$STUDIO_CONCURRENT" ]; then
+ echo "::notice title=Studio shares a card::the Studio GPU half is pinned to one T4 beside a small leg, so this run does NOT cover Studio's own choice between two cards. The report records tensor_split_over_two_cards=false rather than claiming it."
+ fi
+ # A leg list REPLACES --all-kernels rather than filtering it: the
+ # kernel plan and the payload count both come out of the same call,
+ # so a filter applied afterwards would leave the launcher expecting
+ # payloads that were never built.
+ if [ -n "$LEG_LIST" ]; then
+ KERNEL_SELECT="--legs $LEG_LIST"
+ echo "::notice title=Leg override::running $LEG_LIST instead of the wired set"
+ else
+ KERNEL_SELECT="--all-kernels"
+ fi
python .github/scripts/kaggle_t4_ci/build_kernel.py \
--payload-dir tests/kaggle/t4_smoke \
--out kernel \
- --all-kernels \
+ $KERNEL_SELECT \
--unsloth-ref '${{ steps.ref.outputs.ref }}' \
--zoo-ref '${{ steps.pins.outputs.zoo_ref }}' \
--smoke-args "--max-steps $MAX_STEPS" \
- --per-run-timeout 3300 \
+ --with-studio \
+ --studio-args "--max-steps 8" \
+ --per-run-timeout 3900 \
+ $STUDIO_CONCURRENT \
+ $SHARED_WHEELS \
$SKIP
# kernel-timeout-sec is passed to Kaggle at push time as a BACKSTOP, not
@@ -857,6 +1016,21 @@ jobs:
--poll-every 60 \
--deadline-epoch "$(( JOB_START_EPOCH + JOB_TIMEOUT_MINUTES * 60 ))"
+ # The launcher collects executed notebooks and the kernel log and nothing
+ # else, deliberately. The Studio payload therefore encodes its
+ # screenshots and logs into its own cell output; this puts them back.
+ # Lifted from kaggle-t4-studio-gpu-ci.yml along with the payload itself.
+ #
+ # It is safe on a run that carried no Studio payload: the collector finds
+ # no encoded evidence and writes nothing.
+ - name: Unpack the Playwright evidence
+ if: always() && steps.recheck.outputs.should_run == 'true'
+ continue-on-error: true
+ run: |
+ python .github/scripts/kaggle_studio_ci/collect_evidence.py \
+ --evidence kaggle_evidence \
+ --outdir studio_evidence
+
# always(): the evidence is most valuable when something went wrong, and
# the executed notebooks are kilobytes.
#
@@ -881,6 +1055,7 @@ jobs:
name: kaggle-t4-evidence
path: |
kaggle_evidence/**
+ studio_evidence/**
kernel*.ipynb
# 14 rather than the repo's usual 7: this job runs on a minority of
# commits, so two consecutive samples can be a fortnight apart, and
@@ -905,4 +1080,18 @@ jobs:
run: |
python .github/scripts/kaggle_t4_ci/report.py \
--evidence kaggle_evidence \
- --expect '${{ steps.build.outputs.payloads || 4 }}'
+ --expect '${{ steps.build.outputs.legs || 4 }}'
+
+ # Two reporters over ONE evidence directory, each rendering the labels it
+ # owns: the four training legs above, the Studio assertions here. They
+ # are not merged because the two payload shapes are not comparable -- a
+ # per-step metric trace against a list of assertions -- and a single
+ # renderer over both produces a training leg made of question marks.
+ #
+ # Its exit code is the same contract as the T4 reporter's: red only for a
+ # payload that ran and failed its assertions.
+ - name: Report Studio
+ if: always() && steps.recheck.outputs.should_run == 'true'
+ run: |
+ python .github/scripts/kaggle_studio_ci/report.py \
+ --evidence kaggle_evidence
diff --git a/.github/workflows/kaggle-t4-studio-gpu-ci.yml b/.github/workflows/kaggle-t4-studio-gpu-ci.yml
index 04f7f3ef6d4..d9628e7c08b 100644
--- a/.github/workflows/kaggle-t4-studio-gpu-ci.yml
+++ b/.github/workflows/kaggle-t4-studio-gpu-ci.yml
@@ -377,8 +377,13 @@ jobs:
- name: Build the kernel notebook
env:
MAX_STEPS: ${{ inputs.max_steps || '8' }}
- CHAT_MODEL: ${{ inputs.chat_model || 'unsloth/Qwen3.5-2B-GGUF' }}
- TRAIN_MODEL: ${{ inputs.train_model || 'unsloth/Qwen2.5-0.5B-Instruct' }}
+ # MTP-GGUF, not the plain GGUF: multi-token prediction is a distinct
+ # serving path in llama.cpp and the plain repo does not exercise it.
+ CHAT_MODEL: ${{ inputs.chat_model || 'unsloth/Qwen3.5-2B-MTP-GGUF' }}
+ # Qwen3.5-2B rather than Qwen2.5-0.5B-Instruct, so Studio trains the
+ # model this CI actually cares about. Confirmed loadable on a T4 in
+ # 4bit by unsloth-probe-vision-recon-c76ea3.
+ TRAIN_MODEL: ${{ inputs.train_model || 'unsloth/Qwen3.5-2B' }}
SKIP_UI: ${{ inputs.skip_ui }}
run: |
ARGS="--max-steps $MAX_STEPS --chat-model $CHAT_MODEL --train-model $TRAIN_MODEL"
diff --git a/tests/kaggle/studio_gpu/run_studio_gpu.py b/tests/kaggle/studio_gpu/run_studio_gpu.py
index cc6232d61c3..84686e099aa 100644
--- a/tests/kaggle/studio_gpu/run_studio_gpu.py
+++ b/tests/kaggle/studio_gpu/run_studio_gpu.py
@@ -66,6 +66,8 @@ import base64
import io
import json
import os
+import re
+import secrets as secrets_module
import shutil
import subprocess
import sys
@@ -193,6 +195,65 @@ def nvidia_used_mib() -> float | None:
return total if seen else None
+def cli_run_gpu_failure(
+ apps_before: dict[int, int] | None,
+ apps_after: dict[int, int] | None,
+ baseline: float | None,
+ settled: float | None,
+) -> tuple[str | None, dict]:
+ """Did a model actually reach the card? Returns (failure or None, detail).
+
+ WHICH RULER, and this one has now been wrong in both directions.
+
+ The device total is a SHARED reading. On kernel
+ unsloth-probe-studio-full2-815a0c it was sampled too early and read 0.0 MiB
+ on a server that did have the weights; the fix was to sample after a served
+ completion. On unsloth-probe-full-concurrent-417238 it read **-182.0 MiB**
+ -- and the same report carried `compute_apps {"6841": 2628}`, so the model
+ was on the card and 2.6 GB of it. That run was the first with
+ --studio-concurrent, so a training leg shared the card and freed memory
+ inside the window. A shared counter cannot attribute, and subtracting two of
+ its samples is not a measurement of THIS process.
+
+ So the verdict comes off the pids that APPEARED during the window, which is
+ per-process and immune to a co-tenant. A pid already on the card before the
+ launch is excluded, or a co-tenant holding gigabytes would satisfy the claim
+ on its own -- which is the same failure in a new costume.
+
+ The device delta is still recorded, and is still the fallback for an
+ nvidia-smi that answers a total but cannot enumerate processes.
+ """
+ detail: dict = {}
+ if baseline is not None and settled is not None:
+ detail["vram_delta_mib"] = round(settled - baseline, 1)
+
+ if apps_after is None:
+ if baseline is None or settled is None:
+ return "nvidia-smi did not answer, so GPU use is unmeasured", detail
+ if settled - baseline < 200.0:
+ return (
+ f"device VRAM grew by {settled - baseline:.1f} MiB across the "
+ f"launch and a served completion, and nvidia-smi could not "
+ f"enumerate processes to attribute it -- `unsloth run` served "
+ f"from the CPU"
+ ), detail
+ return None, detail
+
+ before = apps_before or {}
+ appeared = {pid: mib for pid, mib in apps_after.items() if pid not in before}
+ detail["compute_apps_appeared"] = appeared
+ grew = sum(appeared.values())
+ detail["process_vram_mib"] = grew
+ if grew < 200.0:
+ return (
+ f"no process appeared on the GPU holding more than {grew} MiB "
+ f"across the launch and a served completion (before "
+ f"{sorted(before)}, after {sorted(apps_after)}) -- `unsloth run` "
+ f"served from the CPU"
+ ), detail
+ return None, detail
+
+
def nvidia_compute_apps() -> dict[int, int] | None:
proc = run(
["nvidia-smi", "--query-compute-apps=pid,used_gpu_memory", "--format=csv,noheader,nounits"],
@@ -203,14 +264,60 @@ def nvidia_compute_apps() -> dict[int, int] | None:
return parse_compute_apps(proc.stdout)
+def visible_device_indices() -> list[int] | None:
+ """The physical card indices CUDA_VISIBLE_DEVICES exposes, or None if unset.
+
+ An empty string is a deliberate "no cards", which is different from unset
+ and must not read as "all of them".
+ """
+ raw = os.environ.get("CUDA_VISIBLE_DEVICES")
+ if raw is None:
+ return None
+ out: list[int] = []
+ for part in raw.split(","):
+ part = part.strip()
+ if not part:
+ continue
+ try:
+ out.append(int(part))
+ except ValueError:
+ # A UUID form (GPU-xxxx) selects a card this cannot map to an
+ # nvidia-smi row index. Returning None would report every card as
+ # usable, which is the failure being fixed, so an unparseable entry
+ # counts as one card rather than as all of them.
+ out.append(-1)
+ return out
+
+
def gpu_inventory() -> list[str]:
+ """The cards THIS PROCESS can use, not the cards the box has.
+
+ `nvidia-smi` enumerates PHYSICAL devices and ignores CUDA_VISIBLE_DEVICES,
+ and reading it as "what is available" produced a false claim on kernel
+ unsloth-probe-full-concurrent-417238. build_kernel.py pins every payload
+ with `CUDA_VISIBLE_DEVICES = str(gpu_index)`, and under --studio-concurrent
+ that includes Studio -- so the run recorded `cards_visible: 2` and
+ `tensor_split_over_two_cards: True` for a server that had ONE card, and sent
+ `tensor_split: [1.0, 1.0]` asking llama.cpp to split across a device that
+ was not there. It loaded anyway, so the assertion passed green.
+
+ That field exists precisely to stop a check keeping its name while testing
+ less. It was sized from the wrong instrument and did exactly that.
+ """
proc = run(
["nvidia-smi", "--query-gpu=name,memory.total,compute_cap", "--format=csv,noheader"],
timeout = 60,
)
if proc.returncode != 0:
return []
- return [line.strip() for line in proc.stdout.splitlines() if line.strip()]
+ rows = [line.strip() for line in proc.stdout.splitlines() if line.strip()]
+ visible = visible_device_indices()
+ if visible is None:
+ return rows
+ # Index into the physical rows where we can; an index nvidia-smi does not
+ # have, or the UUID sentinel, still counts as one card so the COUNT stays
+ # right even when the description cannot be recovered.
+ return [rows[i] if 0 <= i < len(rows) else "visible GPU (details unavailable)" for i in visible]
def log_paths(server_log: Path, studio_home: Path) -> list[Path]:
@@ -322,10 +429,44 @@ class Payload:
# artifact", so every log that leaves this machine is scrubbed of it.
self.secrets: set[str] = set()
+ # Resolved and registered HERE, before anything can log it. `auto`
+ # mints a fresh one per run rather than carrying a constant, because a
+ # constant in a repo is a credential whether or not it is ever reachable.
+ if self.args.studio_password == "auto":
+ self.args.studio_password = "ci-" + secrets_module.token_urlsafe(18)
+ if self.args.studio_password:
+ self.secrets.add(self.args.studio_password)
+
# ---------------------------------------------------------------- report
def record(self, name: str, passed: bool, detail: dict) -> bool:
- entry = {"name": name, "passed": bool(passed), **detail}
+ # Per-assertion wall clock, because this payload is now the longest
+ # thing in the kernel and nothing said where the time went. Measured on
+ # unsloth-probe-full-concurrent-417238: 1487.5s across 19 assertions,
+ # and the report carried no breakdown at all, so every question about
+ # shortening Studio was guesswork.
+ #
+ # It is elapsed-since-the-previous-record, NOT a timer around the
+ # assertion body, and the name says so. The assertions run back to
+ # back, so the two are the same to within the bookkeeping between them;
+ # the first entry measures from process start, which is the setup
+ # before any assertion and is worth seeing rather than hiding.
+ #
+ # Both reads are `getattr` with a default: `record` is driven directly
+ # by several CPU guards against stub objects that have no clock, and a
+ # hard `self.started` turns every one of them into an AttributeError
+ # about timing instead of a result about the rule they test.
+ now = time.time()
+ started = getattr(self, "started", now)
+ previous = getattr(self, "_last_record_at", started)
+ self._last_record_at = now
+ entry = {
+ "name": name,
+ "passed": bool(passed),
+ "seconds_since_previous": round(now - previous, 1),
+ "at_seconds": round(now - started, 1),
+ **detail,
+ }
self.assertions.append(entry)
for reason in detail.get("failures", []) or []:
self.failures.append(f"{name}: {reason}")
@@ -431,7 +572,24 @@ class Payload:
break
else:
head = [sys.executable, "-c", "from unsloth_cli import app; app()"]
- return head + ["studio", "-H", "127.0.0.1", "-p", str(self.args.port)]
+ cmd = head + ["studio", "-H", "127.0.0.1", "-p", str(self.args.port)]
+ if self.args.studio_password:
+ # The HEADLESS path, and it is a feature rather than a convenience
+ # here: `--password` sets the INITIAL admin password when none is
+ # set yet, which is exactly the shape a server started by a script
+ # is in. Without it the only way in is the bootstrap password
+ # Studio seeds into a file and prints to its own log, and a
+ # deployment that has to read a log to log in is not one anybody
+ # scripts twice.
+ #
+ # The value is generated per run and registered as a secret before
+ # this is ever called, so it is scrubbed out of every log and
+ # evidence bundle that leaves the machine. It is still visible in
+ # this session's process list, which the flag's own help says; that
+ # is acceptable for a single-tenant CI kernel and would not be on a
+ # shared host.
+ cmd += ["--password", self.args.studio_password]
+ return cmd
def start_server(self) -> bool:
# An absent auth directory is what re-seeds the bootstrap password;
@@ -504,8 +662,25 @@ class Payload:
return self.scrub(" | ".join(text.splitlines()[-lines:]))
def authenticate(self) -> bool:
- path = self.studio_home / "auth" / ".bootstrap_password"
failures: list[str] = []
+ if self.args.studio_password:
+ # Log in with the password we PASSED, which is the assertion: if
+ # --password had been ignored, Studio would have seeded a bootstrap
+ # password instead and this login would fail. A run that fell back
+ # to the bootstrap on failure would pass while proving the flag does
+ # nothing, so there is deliberately no fallback.
+ try:
+ self.studio.login(self.args.studio_password)
+ except StudioError as exc:
+ failures.append(
+ f"--password was passed to `unsloth studio` and logging in "
+ f"with it failed, so the flag did not take effect: {exc}"
+ )
+ return self.record(
+ "authenticate", not failures, {"source": "--password", "failures": failures}
+ )
+
+ path = self.studio_home / "auth" / ".bootstrap_password"
if not path.is_file():
failures.append(f"no bootstrap password was seeded at {path}")
return self.record("authenticate", False, {"failures": failures})
@@ -517,7 +692,9 @@ class Payload:
self.studio.login(password)
except StudioError as exc:
failures.append(str(exc))
- return self.record("authenticate", not failures, {"failures": failures})
+ return self.record(
+ "authenticate", not failures, {"source": "bootstrap", "failures": failures}
+ )
def server_alive(self) -> bool:
return self.proc is not None and self.proc.poll() is None
@@ -646,6 +823,19 @@ class Payload:
f"(a clamp to the model's layer count looks like this)"
)
+ # Speculative decoding, read off the status rather than inferred from
+ # the repo name. Pointing --chat-model at an MTP repo does NOT prove the
+ # drafter engaged: if the companion is missing, llama.cpp was built
+ # without MTP, or the drafter was downgraded for VRAM, the main GGUF
+ # loads and generates exactly as it does otherwise and every assertion
+ # here stays green. `spec_fallback_reason` is what names which of those
+ # happened, so it is recorded whether or not it fires.
+ spec = {
+ "drafter_kind": status_body.get("spec_drafter_kind"),
+ "fallback_reason": status_body.get("spec_fallback_reason"),
+ "supports_mtp": status_body.get("llama_cpp_supports_mtp"),
+ }
+
return {
"model": model_path,
"variant": variant,
@@ -653,6 +843,7 @@ class Payload:
"effective_gpu_layers": effective,
"llama_server_pids": server_pids,
"device_vram_delta_mib": None if delta is None else round(delta, 1),
+ "spec_decoding": spec,
"evidence": verdict["evidence"],
"positives": verdict["positives"],
"failures": failures,
@@ -688,8 +879,368 @@ class Payload:
detail["failures"].append(f"/v1/chat/completions returned HTTP {code}")
elif not text.strip():
detail["failures"].append("the model on the GPU returned empty content")
+
+ # The MTP claim, checked rather than assumed from the repo name.
+ # An MTP repo whose drafter never engaged serves ordinary decoding
+ # and passes everything above it, so selecting the repo is a request
+ # and this is the result. The reason is named rather than reported
+ # as "MTP off", because "llama.cpp has no MTP support" and "the
+ # drafter was downgraded for VRAM" are different findings and only
+ # one of them is about this leg.
+ spec = detail.get("spec_decoding") or {}
+ if "MTP" in (self.args.chat_model or "").upper() and spec.get("drafter_kind") != "mtp":
+ detail["failures"].append(
+ f"--chat-model is {self.args.chat_model}, which was chosen to "
+ f"exercise multi-token prediction, and Unsloth reports "
+ f"spec_drafter_kind={spec.get('drafter_kind')!r} "
+ f"(fallback_reason={spec.get('fallback_reason')!r}, "
+ f"llama_cpp_supports_mtp={spec.get('supports_mtp')!r}). The "
+ f"model served ordinary decoding, so nothing here covers the "
+ f"MTP path and pointing at the MTP repo bought nothing"
+ )
return self.record("gpu_inference", not detail["failures"], detail)
+ def assert_server_flags(self) -> bool:
+ """Reload with a quantized KV cache and a two-card split, and check what
+ was APPLIED rather than what was asked for.
+
+ Studio's own status distinguishes the two, which is the point: it
+ carries `cache_type_kv` for the live server alongside a coverage field
+ and a reason for anything it could not apply. So a request that was
+ silently dropped is visible, and a check that only asserted "the load
+ succeeded" would pass on exactly that.
+
+ `tensor_split` over two T4s is the flag the brief asks about and the
+ one nothing here has ever exercised. The split is sized to the cards
+ that are VISIBLE, because --studio-concurrent pins this half to one
+ card so it can share with a training leg, and the report says which of
+ the two it did: a single-card run records
+ `tensor_split_over_two_cards: false` with a note rather than passing
+ under the same name. A check that keeps its name while quietly testing
+ less is the failure this file exists against.
+
+ The context length is pinned to `--studio-ctx` (2048 by default) rather
+ than left at the model default, because an unconstrained context on a
+ 14.56GB card is how a KV-cache test turns into an OOM about something
+ else.
+ """
+ failures: list[str] = []
+ cards = gpu_inventory()
+ detail: dict = {"requested": {}, "cards_visible": len(cards)}
+ body = {
+ "model_path": self.args.chat_model,
+ "is_lora": False,
+ "max_seq_length": self.args.studio_ctx,
+ "gpu_memory_mode": "manual",
+ "gpu_layers": self.args.gpu_layers,
+ "force": True,
+ # q8_0 rather than a fancier width: it is the one every llama.cpp
+ # build supports, so a refusal here is about the MODEL's cache
+ # layout and not about the binary.
+ "cache_type_kv": "q8_0",
+ # One weight per VISIBLE card, even. The values are relative
+ # weights, not byte counts.
+ #
+ # Sized rather than hardcoded to two, because --studio-concurrent
+ # pins this half to a single card so it can share with a training
+ # leg: build_kernel.py's run_one sets CUDA_VISIBLE_DEVICES to the
+ # card it was admitted on. Sending [1.0, 1.0] to a one-card server
+ # asks llama.cpp to split across a device that is not there, and
+ # what comes back is a failure about the load rather than about the
+ # flag.
+ "tensor_split": [1.0] * max(1, len(cards)),
+ }
+ if self.args.chat_variant:
+ body["gguf_variant"] = self.args.chat_variant
+ detail["requested"] = {
+ k: body[k] for k in ("max_seq_length", "cache_type_kv", "tensor_split")
+ }
+ # STATED, not silent. Under --studio-concurrent this half runs on one
+ # card so it can share with a training leg, and a split over one device
+ # is not the two-card flag the brief asks about. Recording it as
+ # exercised when it was not is how a check keeps its name and loses its
+ # meaning; a reader of this report can see which machine it ran on.
+ detail["tensor_split_over_two_cards"] = len(cards) >= 2
+ if len(cards) < 2:
+ detail["tensor_split_note"] = (
+ f"only {len(cards)} card visible, so the two-card tensor_split "
+ f"was NOT exercised; the KV-cache type, the context pin and the "
+ f"GPU residency below still were"
+ )
+
+ try:
+ self.studio.expect("POST", "/api/inference/load", body, timeout = self.args.load_timeout)
+ except StudioError as exc:
+ failures.append(f"loading with server flags failed: {exc}"[:600])
+ detail["failures"] = failures
+ return self.record("server_flags", False, detail)
+
+ code, status_body = self.studio.get("/api/inference/status")
+ detail["status_code"] = code
+ status = status_body if isinstance(status_body, dict) else {}
+ applied = {
+ k: status.get(k)
+ for k in (
+ "cache_type_kv",
+ "context_length",
+ "max_context_length",
+ "gpu_layers",
+ "is_mlx",
+ )
+ }
+ detail["applied"] = applied
+
+ # 1. the KV cache type actually in force. Reported rather than asserted
+ # equal: a model whose cache layout cannot be quantized is entitled
+ # to refuse, and Studio says so. What is NOT acceptable is silence.
+ got_cache = (applied.get("cache_type_kv") or "").lower()
+ if not got_cache:
+ failures.append(
+ "the status reports no cache_type_kv at all after a load that "
+ "asked for q8_0, so there is no way to tell whether the request "
+ "was honoured, downgraded or ignored"
+ )
+ elif got_cache != "q8_0":
+ # Not a failure by itself -- but it must come with a reason.
+ reason = status.get("kv_quant_reason") or status.get("mlx_kv_quant_reason")
+ detail["cache_downgrade_reason"] = reason
+ if not reason:
+ failures.append(
+ f"q8_0 was requested and {got_cache!r} is in force, with no "
+ f"reason given. A silent downgrade is the failure this check "
+ f"exists for"
+ )
+
+ # 2. the context length is the one that was pinned. llama-server admits
+ # a prompt on n_ctx alone, so a server running at the model default
+ # behaves differently from one at 2048 and the difference is
+ # invisible in a chat response.
+ ctx = applied.get("context_length")
+ if ctx is None:
+ failures.append("the status reports no context_length")
+ elif int(ctx) > self.args.studio_ctx:
+ failures.append(
+ f"context_length is {ctx}, above the {self.args.studio_ctx} that "
+ f"was requested, so the pin did not take"
+ )
+
+ # 3. it is still on the GPU. A tensor split that fell back to CPU would
+ # otherwise report a healthy server and prove nothing about either
+ # card.
+ used = nvidia_used_mib()
+ detail["gpu_used_mib"] = used
+ if used is not None and used < 200:
+ failures.append(
+ f"only {used} MiB of GPU memory is in use after a "
+ f"{len(cards)}-card tensor_split load, so this is running on "
+ f"the CPU"
+ )
+
+ detail["failures"] = failures
+ return self.record("server_flags", not failures, detail)
+
+ def assert_compaction(self) -> bool:
+ """A conversation past the window must COMPACT, and a short one must not.
+
+ The context length is pinned to `--studio-ctx` by `assert_server_flags`
+ immediately above, so this is the one place the payload knows what the
+ window is and can overflow it deliberately.
+
+ Studio reports the fit on the completion itself, as `context_truncated`
+ with `dropped_messages`, so the claim is readable without a browser.
+ The rule is a PAIR, and the second half is what makes the first mean
+ anything:
+
+ * a conversation built past the budget comes back 200 with
+ `dropped_messages > 0` -- it was shortened, not refused;
+ * a two-message conversation comes back with nothing dropped.
+
+ Asserting only the first passes on a server that reports truncation
+ unconditionally, which is indistinguishable from working and is the
+ failure this file keeps being caught by. Asserting only the second
+ passes on a server that never compacts at all and returns a
+ context-length error instead.
+ """
+ failures: list[str] = []
+ detail: dict = {}
+
+ def _dropped(body) -> int | None:
+ if not isinstance(body, dict):
+ return None
+ truncation = body.get("context_truncated")
+ if not isinstance(truncation, dict):
+ return 0
+ try:
+ return int(truncation.get("dropped_messages") or 0)
+ except (TypeError, ValueError):
+ return 0
+
+ # Distinct filler, so nothing can be deduplicated or cached into
+ # fitting. Sized past `studio_ctx` by a wide margin rather than a
+ # narrow one: the budget subtracts the reply reserve and the template's
+ # own framing, so a prompt that merely equals the context length is not
+ # reliably over it.
+ long_messages = []
+ for i in range(40):
+ long_messages.append(
+ {
+ "role": "user",
+ "content": f"Fact {i}: the {i}th token of this transcript is "
+ + " ".join(f"w{i}x{j}" for j in range(40)),
+ }
+ )
+ long_messages.append({"role": "assistant", "content": f"Noted fact {i}."})
+ long_messages.append({"role": "user", "content": "Reply with one word."})
+
+ # `context_overflow` defaults to "error", and that default is CORRECT:
+ # a conversation past the window comes back 400 with
+ # code=context_length_exceeded so a client's own trim loop can see it.
+ # Compaction is a policy you ask for, and asking for it is the thing
+ # under test. Measured the hard way on kernel
+ # unsloth-probe-studio-full2-815a0c, where this assertion failed a
+ # documented default. "truncate_oldest" is the policy that applies to a
+ # plain chat; "truncate_middle" is limited to client-tool and
+ # response_format passthrough (studio/backend/models/inference.py).
+ code, body = self.chat(long_messages, max_tokens = 32, context_overflow = "truncate_oldest")
+ detail["context_overflow"] = "truncate_oldest"
+ detail["long_status"] = code
+ detail["long_messages"] = len(long_messages)
+ detail["long_truncation"] = (
+ (body or {}).get("context_truncated") if isinstance(body, dict) else None
+ )
+ long_dropped = _dropped(body)
+ if code != 200:
+ failures.append(
+ f"an over-length conversation returned HTTP {code} instead of "
+ f"being compacted: {str(body)[:300]}"
+ )
+ elif not long_dropped:
+ failures.append(
+ f"a {len(long_messages)}-message conversation against a "
+ f"{self.args.studio_ctx}-token window dropped nothing, so no "
+ f"compaction happened and the reply used a prompt nobody sized"
+ )
+
+ # The POLICY control: the same over-length conversation with the
+ # default `context_overflow` must be REFUSED. Without this, the check
+ # above passes on a server that compacts everything regardless of what
+ # was asked for, and the field name in the request would be decorative.
+ code, body = self.chat(long_messages, max_tokens = 32)
+ detail["long_status_default_policy"] = code
+ # The CODE, not just the 400. `openai_error_body` always emits the key
+ # and leaves it None for an unclassified failure, so a validation error
+ # or a code-less generation error is also a 400 and would satisfy a
+ # status-only check while proving nothing about the overflow semantics
+ # this failure text promises.
+ refusal_code = None
+ if isinstance(body, dict):
+ refusal_code = (body.get("error") or {}).get("code")
+ detail["long_default_policy_code"] = refusal_code
+ if code != 400 or refusal_code != "context_length_exceeded":
+ failures.append(
+ f"with the default context_overflow the same over-length "
+ f"conversation returned HTTP {code} with error code "
+ f"{refusal_code!r}, not 400 with code=context_length_exceeded. "
+ f"That code is what a client's own trim loop detects, and if "
+ f"everything is compacted anyway then asking for "
+ f"truncate_oldest above proved nothing"
+ )
+
+ # The LENGTH control, and it is not optional either: without it a
+ # server that always claims truncation passes the first check.
+ code, body = self.chat([{"role": "user", "content": "Say hi."}], max_tokens = 16)
+ detail["short_status"] = code
+ short_dropped = _dropped(body)
+ detail["short_dropped"] = short_dropped
+ if code != 200:
+ failures.append(f"the short control returned HTTP {code}")
+ elif short_dropped:
+ failures.append(
+ f"a two-message conversation reported {short_dropped} dropped "
+ f"messages, so the field is not responsive to length and the "
+ f"check above proves nothing"
+ )
+
+ detail["failures"] = failures
+ return self.record("compaction", not failures, detail)
+
+ def assert_api_key(self) -> bool:
+ """Mint an API key and DRIVE it, which is the half that can be wrong.
+
+ Creating a key proves the endpoint returns a string. Whether that
+ string authenticates anything is a separate question, and the answer
+ that matters: an API key Studio issues and then rejects is worse than
+ no API key, because the failure surfaces in a user's integration rather
+ than here.
+
+ Three claims, in order of what each rules out:
+
+ 1. the key is minted and returned ONCE (the raw value is not
+ retrievable later, by design);
+ 2. a request carrying ONLY that key succeeds -- the session bearer
+ token is set aside for the call, or this would pass on the token and
+ say nothing about the key;
+ 3. a request carrying a corrupted key FAILS. Without that, a server
+ that ignores the header entirely passes claim 2.
+ """
+ failures: list[str] = []
+ detail: dict = {}
+ raw_key = None
+ try:
+ code, body = self.studio.post(
+ "/api/auth/api-keys", {"name": f"kaggle-ci-{int(time.time())}"}
+ )
+ detail["create_status"] = code
+ if code >= 400 or not isinstance(body, dict):
+ failures.append(f"could not create an API key: {code} {str(body)[:200]}")
+ else:
+ raw_key = body.get("key")
+ detail["has_key"] = bool(raw_key)
+ detail["key_metadata"] = {
+ k: v
+ for k, v in (body.get("api_key") or {}).items()
+ if k in ("id", "name", "is_active", "expires_at")
+ }
+ if not raw_key:
+ failures.append("the create response carried no raw key")
+ except BaseException as exc: # noqa: BLE001
+ failures.append(f"creating an API key raised: {type(exc).__name__}: {exc}"[:300])
+
+ if raw_key:
+ # Registered before it is used anywhere, so it cannot reach a log.
+ self.secrets.add(raw_key)
+ saved = self.studio.token
+ try:
+ # The key AS the bearer, with the session token set aside. If
+ # the session token were left in place this would pass on the
+ # token and prove nothing.
+ self.studio.token = raw_key
+ code, body = self.studio.get("/api/auth/api-keys")
+ detail["key_auth_status"] = code
+ if code >= 400:
+ failures.append(
+ f"the API key Studio just issued does not authenticate: "
+ f"{code} {str(body)[:200]}"
+ )
+
+ # And a corrupted key must be REJECTED. Without this, a server
+ # that ignores the header entirely passes the check above.
+ self.studio.token = raw_key[:-4] + "0000" if len(raw_key) > 8 else "bogus"
+ code, _ = self.studio.get("/api/auth/api-keys")
+ detail["bad_key_status"] = code
+ if code < 400:
+ failures.append(
+ f"a corrupted API key was accepted ({code}), so the "
+ f"check above passes whatever is sent"
+ )
+ except BaseException as exc: # noqa: BLE001
+ failures.append(f"driving the API key raised: {type(exc).__name__}: {exc}"[:300])
+ finally:
+ self.studio.token = saved
+
+ detail["failures"] = failures
+ return self.record("api_key", not failures, detail)
+
def assert_tool_calling(self) -> bool:
failures: list[str] = []
detail: dict = {}
@@ -733,6 +1284,198 @@ class Payload:
detail["failures"] = failures
return self.record("tool_calling", not failures, detail)
+ def assert_code_execution(self) -> bool:
+ """The python tool must RUN, and the proof is a file on disk.
+
+ `assert_tool_calling` above proves the model can EMIT a call. That is a
+ different claim: a weather tool is never executed by Studio at all, the
+ caller is expected to run it. The local `python` tool is executed by
+ Studio itself, in a per-session sandbox, and the interesting failure is
+ the loop offering the tool and never running it -- which looks
+ identical from the response text, because the model will happily
+ narrate a result it never received.
+
+ So the evidence is not the prose. A token is minted here, the model is
+ asked to write it to a file, and this reads it back out of
+ `<studio home>/sandbox`, where `sandbox_root()` puts the per-session
+ working directories. Nothing in the reply can fake that; only an
+ executed `open(...).write(...)` puts those bytes on this disk.
+
+ Two settings are not incidental and must not be "simplified":
+
+ * `permission_mode = "off"`. `routes/inference.py` REJECTS a local
+ python/terminal tool under `ask`, and under `auto` or an omitted
+ default, with a 400 -- there is no confirmation channel here. The
+ run would fail on configuration and look like a broken tool.
+ * `tool_choice` is left alone. Forcing it would prove the schema is
+ reachable, not that the loop runs what it selected, and the file is
+ the claim either way.
+
+ The filename is not asserted, only the CONTENT: a small local model
+ rewording a path is not a Studio defect, and any file carrying the
+ token was written by code that ran.
+ """
+ failures: list[str] = []
+ detail: dict = {}
+
+ token = "unslothcodeexec" + secrets_module.token_hex(8)
+ sandbox = self.studio_home / "sandbox"
+ detail["sandbox_root"] = str(sandbox)
+ before = set(sandbox.rglob("*")) if sandbox.exists() else set()
+
+ code, payload = self.chat(
+ [
+ {
+ "role": "user",
+ "content": (
+ "Use the python tool to run exactly this code, then reply "
+ "with the single word done:\n\n"
+ f"open({token + '.txt'!r}, 'w').write({token!r})"
+ ),
+ }
+ ],
+ enable_tools = True,
+ enabled_tools = ["python"],
+ permission_mode = "off",
+ max_tokens = 512,
+ )
+ detail["http_status"] = code
+ if code != 200 or not isinstance(payload, dict):
+ failures.append(
+ f"the code-execution completion returned HTTP {code}: {str(payload)[:300]}"
+ )
+ else:
+ choice = (payload.get("choices") or [{}])[0]
+ message = choice.get("message") or {}
+ detail["finish_reason"] = choice.get("finish_reason")
+ detail["reply"] = (message.get("content") or "")[:300]
+
+ # The claim, read off the filesystem rather than off the reply.
+ written = []
+ if sandbox.exists():
+ for path in sandbox.rglob("*"):
+ if not path.is_file():
+ continue
+ try:
+ body = path.read_text(encoding = "utf-8", errors = "replace")
+ except OSError:
+ continue
+ if token in body or token in path.name:
+ written.append(str(path.relative_to(sandbox)))
+ detail["files_carrying_the_token"] = written
+ detail["new_sandbox_entries"] = sorted(
+ str(p.relative_to(sandbox))
+ for p in ((set(sandbox.rglob("*")) - before) if sandbox.exists() else set())
+ )[:20]
+
+ if not written:
+ failures.append(
+ "no file under the sandbox carries the token, so the python "
+ "tool was offered but never executed -- whatever the reply says"
+ )
+
+ detail["failures"] = failures
+ return self.record("code_execution", not failures, detail)
+
+ def assert_web_search(self) -> bool:
+ """The web_search tool must be EXECUTED, and the log is where that shows.
+
+ Same shape as `assert_code_execution` and a different instrument,
+ because a web search leaves nothing on disk to read back. Studio logs
+ `execute_tool: name=...` from INSIDE `execute_tool`, so the line is
+ emitted by execution rather than by selection -- a loop that offered
+ the tool and never ran it produces no such line, which is the failure
+ worth catching.
+
+ What is deliberately NOT asserted: that the search returned results.
+ `_web_search` fans out through ddgs with no API key, and a provider
+ rate-limiting a Kaggle egress IP is a fact about the day, not a Studio
+ defect. Failing on it would put a red in front of every PR for
+ something no reader could act on. The reply and the result count are
+ recorded so a human can see which happened.
+ """
+ failures: list[str] = []
+ detail: dict = {}
+
+ # TWO attempts, and the second is what makes a failure diagnosable.
+ # `enabled_tools = ["web_search"]` is one name out of ALL_TOOLS, and
+ # `routes/inference.py` also reads a request naming only hosted-tool
+ # names as a provider-hosted ask. Omitting `enabled_tools` selects
+ # every local tool instead, which is a different path through the same
+ # loop. If the first fails and the second executes, the fault is in the
+ # single-name selection; if neither does, the model will not call the
+ # tool however it is offered. Reporting "the loop offered web_search
+ # and never ran it" off one attempt was a guess dressed as a finding:
+ # nothing in that run showed the tool had been offered at all.
+ marker = "execute_tool: name=web_search"
+ prompt = (
+ "Search the web for the current version of the Linux kernel, "
+ "then answer in one sentence."
+ )
+ attempts: list[dict] = []
+ for label, selection in (
+ ("named", {"enabled_tools": ["web_search"]}),
+ ("all_local_tools", {}),
+ ):
+ before = ""
+ try:
+ before = self.server_log.read_text(encoding = "utf-8", errors = "replace")
+ except OSError:
+ pass
+
+ code, payload = self.chat(
+ [{"role": "user", "content": prompt}],
+ enable_tools = True,
+ permission_mode = "off",
+ # Forced BY NAME. `tool_choice: "required"` was tried first and
+ # changed nothing -- kernel unsloth-probe-studio-r3-0b85d4
+ # returned the same parametric answer, "The current version of
+ # the Linux kernel is 6.10", with executions 0. A bare
+ # "required" says only that some tool must be called; the dict
+ # form pins the function, and
+ # `chat_template_helpers.forced_tool_name` reads exactly this
+ # shape. Without a force this measures whether a 2B model
+ # DECIDES to search, which is a model property rather than a
+ # Studio one.
+ tool_choice = {"type": "function", "function": {"name": "web_search"}},
+ max_tokens = 512,
+ **selection,
+ )
+ record = {"selection": label, "http_status": code}
+ if code != 200 or not isinstance(payload, dict):
+ record["error"] = str(payload)[:200]
+ else:
+ choice = (payload.get("choices") or [{}])[0]
+ record["finish_reason"] = choice.get("finish_reason")
+ record["reply"] = ((choice.get("message") or {}).get("content") or "")[:200]
+
+ try:
+ after = self.server_log.read_text(encoding = "utf-8", errors = "replace")
+ except OSError:
+ after = ""
+ # Only what THIS request wrote, so an earlier attempt's tool call
+ # cannot be read as this one's evidence.
+ fresh = after[len(before) :]
+ record["executions"] = fresh.count(marker)
+ # Any tool at all, which separates "the loop ran and chose
+ # something else" from "the loop never ran".
+ record["any_tool_executions"] = fresh.count("execute_tool: name=")
+ attempts.append(record)
+ if record["executions"]:
+ break
+
+ detail["attempts"] = attempts
+ detail["executions"] = sum(a["executions"] for a in attempts)
+ if not detail["executions"]:
+ failures.append(
+ f"{marker!r} never appeared in the server log for either "
+ f"selection, so web_search was not executed however it was "
+ f"offered: {attempts}"
+ )
+
+ detail["failures"] = failures
+ return self.record("web_search", not failures, detail)
+
# ----------------------------------------------------------- assertion B
def assert_training(self) -> bool:
@@ -1063,8 +1806,664 @@ class Payload:
detail["failures"] = failures
return self.record("gguf_export", not failures, detail)
+ # Every tab in Studio, and the backing endpoint its first render calls.
+ # A tab whose router failed to import does not render a broken panel; the
+ # route is simply absent and the request 404s, which is what this looks for.
+ # Named by tab so a failure says which one, and kept as a mapping so a new
+ # tab that is not covered here is visible as an omission rather than
+ # invisible.
+ TAB_ENDPOINTS = (
+ # NOT /jobs/current: that legitimately 404s when no job is running,
+ # so it would be red on correct behaviour. This one is unconditional
+ # and lives in the same router package, so an import failure takes it
+ # down with everything else.
+ ("data_designer", "/api/data-recipe/seed/github/env-token"),
+ ("image_creation", "/api/inference/images/status"),
+ ("image_creation_gallery", "/api/inference/images/gallery"),
+ ("video_creation", "/api/inference/video/status"),
+ ("video_creation_gallery", "/api/inference/video/gallery"),
+ ("image_training", "/api/train/diffusion/status"),
+ ("image_training_runs", "/api/train/diffusion/runs"),
+ ("training", "/api/train/status"),
+ ("models", "/api/models/local"),
+ ("datasets", "/api/datasets/local"),
+ )
+
+ def assert_tabs(self) -> bool:
+ """Every tab's backing endpoint answers.
+
+ This is a smoke and it is honest about being one: it does not click
+ through the UI, it asks each tab's own first request. What it catches
+ is the failure that actually happens -- a route module that raised on
+ import, so the router was never mounted and the tab renders an error
+ the moment it is opened. That is invisible to every other assertion
+ here, all of which touch inference, training and export only.
+
+ A 404 is the specific signal, so it is separated from other statuses
+ in the record rather than folded into "not 200": a 500 is a live route
+ with a broken handler and a 404 is a route that does not exist, and
+ they lead a reader to different places.
+
+ An endpoint answering 200 with a non-object is a failure too. Several
+ of these are declared with a `response_model`, so a bare string or null
+ means something upstream is substituting for the real handler.
+ """
+ failures: list[str] = []
+ detail: dict = {"checked": len(self.TAB_ENDPOINTS)}
+ results: dict = {}
+
+ for name, path in self.TAB_ENDPOINTS:
+ try:
+ code, body = self.studio.get(path)
+ except BaseException as exc: # noqa: BLE001
+ results[name] = {"path": path, "error": f"{type(exc).__name__}: {exc}"[:200]}
+ failures.append(f"{name}: {path} raised {type(exc).__name__}")
+ continue
+ entry = {"path": path, "status": code, "type": type(body).__name__}
+ results[name] = entry
+ if code == 404:
+ failures.append(
+ f"{name}: {path} is 404, so its router was never mounted -- "
+ f"the tab renders an error the moment it is opened"
+ )
+ elif code >= 400:
+ failures.append(f"{name}: {path} returned HTTP {code}: {str(body)[:200]}")
+ elif not isinstance(body, (dict, list)):
+ failures.append(
+ f"{name}: {path} answered 200 with {type(body).__name__}, not a "
+ f"JSON object, so something is standing in for the handler"
+ )
+
+ detail["tabs"] = results
+ detail["failures"] = failures
+ return self.record("tabs", not failures, detail)
+
+ def assert_lora_vs_base(self, gguf: str | None) -> bool:
+ """The exported model must answer DIFFERENTLY from the base it came from.
+
+ This is the comparison an export check cannot make on its own. The GGUF
+ assertion proves a file was produced, carries the magic, loads on the
+ GPU and generates -- and every one of those is true of an export that
+ silently merged nothing and shipped the base weights. A no-op merge is
+ the regression here, and it is invisible to file size, to the magic and
+ to "it generated text".
+
+ Greedy decoding at temperature 0 makes it visible: identical weights
+ answer identically, so a difference is the adapter.
+
+ **The determinism control is not optional and comes first.** If the
+ SAME weights, loaded twice, do not reproduce their own answer, then a
+ difference between two models says nothing, and this reports that it
+ could not compare rather than passing on the noise. Two loads of the
+ base, then one of the export: the claim is only made once the
+ instrument has been shown to be steady.
+
+ The canary is REPORTED rather than asserted. Studio's training run is a
+ handful of steps and whether that is enough to learn a specific string
+ is a property of the run length, not of the export path -- asserting it
+ would be a red about training tuning wearing an export label.
+ """
+ failures: list[str] = []
+ detail: dict = {"gguf": gguf}
+ prompt = [{"role": "user", "content": "What is the Unsloth Studio Kaggle canary?"}]
+
+ if not gguf:
+ failures.append(
+ "no exported GGUF to compare: the export assertion did not "
+ "produce one, so there is nothing to hold against the base"
+ )
+ detail["failures"] = failures
+ return self.record("lora_vs_base", False, detail)
+
+ def _say(model: str, variant, label: str) -> tuple[str | None, list]:
+ loaded = self.load_model(model, variant = variant, label = label)
+ if loaded["failures"]:
+ return None, [f"{label} did not load: {loaded['failures'][0]}"]
+ code, payload = self.chat(prompt)
+ if code != 200 or not isinstance(payload, dict):
+ return None, [f"{label} returned HTTP {code} on generation"]
+ choices = payload.get("choices") or [{}]
+ return ((choices[0].get("message") or {}).get("content") or ""), []
+
+ base_one, problems = _say(self.args.chat_model, self.args.chat_variant, "base")
+ failures += problems
+ base_two, problems = _say(self.args.chat_model, self.args.chat_variant, "base_again")
+ failures += problems
+ detail["base_first"] = (base_one or "")[:200]
+ detail["base_second"] = (base_two or "")[:200]
+
+ if base_one is not None and base_two is not None:
+ detail["base_reproduces_itself"] = base_one == base_two
+ if base_one != base_two:
+ failures.append(
+ "the base model did not reproduce its own greedy answer "
+ "across two loads, so a difference against the export "
+ "would be noise rather than the adapter"
+ )
+
+ if not failures:
+ tuned, problems = _say(gguf, None, "exported")
+ failures += problems
+ detail["exported_said"] = (tuned or "")[:200]
+ detail["canary_in_exported"] = CANARY in (tuned or "")
+ detail["canary_in_base"] = CANARY in (base_one or "")
+ if tuned is not None:
+ detail["differs_from_base"] = tuned != base_one
+ if tuned == base_one:
+ failures.append(
+ "the exported model gave the base model's answer "
+ "verbatim, so the adapter reached neither the merge "
+ "nor the GGUF -- which every other export check passes"
+ )
+
+ detail["failures"] = failures
+ return self.record("lora_vs_base", not failures, detail)
+
+ def assert_image_generation(self) -> bool:
+ """The image tab, end to end: load, generate, and download the PNG.
+
+ Last priority and the smallest possible run -- 256x256 (the schema's
+ floor) at 2 steps -- because the claim is that the path executes, not
+ that the picture is good.
+
+ "Nothing errored" is not the check. A diffusion pipeline that fails
+ mid-way still writes a gallery record and still returns 200, and a
+ pipeline whose weights never loaded produces a FLAT image, which is a
+ valid PNG. So the evidence is the file:
+
+ * the bytes start with the PNG magic, so what the download endpoint
+ serves is really a PNG rather than a JSON error with a 200 on it;
+ * the IHDR chunk says 256x256, read out of the header rather than
+ taken from the gallery record -- the record repeats what was asked
+ for, and the file says what was made;
+ * the image is not one flat colour. A pipeline that produced nothing
+ returns a uniform frame, which compresses to almost nothing, so this
+ is checked on the decoded extrema where PIL is available and on a
+ compressed-size floor where it is not. Both are recorded, so a
+ reader can see which one ruled.
+ """
+ import urllib.error
+ import urllib.request
+
+ failures: list[str] = []
+ detail: dict = {"model": self.args.image_model}
+ want = 256
+
+ try:
+ code, body = self.studio.post(
+ "/api/inference/images/load",
+ {"model_path": self.args.image_model},
+ timeout = self.args.export_timeout,
+ )
+ detail["load_status"] = code
+ if code >= 400:
+ failures.append(f"images/load returned HTTP {code}: {str(body)[:300]}")
+ detail["failures"] = failures
+ return self.record("image_generation", False, detail)
+
+ # The load is ASYNCHRONOUS. It answers 200 having only accepted the
+ # request, and generating against it answers 409 "No diffusion
+ # model is loaded." -- measured on kernel
+ # unsloth-probe-studio-r3-0b85d4, where load_status was 200 and
+ # generate_status was 409 twelve lines later. So wait for
+ # `images/status` to say `loaded`, and carry `load-progress` while
+ # waiting: a download that stalls or errors is then reported as
+ # what it is instead of arriving as a generation failure.
+ deadline = time.time() + self.args.export_deadline
+ status: dict = {}
+ progress: dict = {}
+ while time.time() < deadline:
+ _, status_body = self.studio.get("/api/inference/images/status")
+ status = status_body if isinstance(status_body, dict) else {}
+ if status.get("loaded"):
+ break
+ _, progress_body = self.studio.get("/api/inference/images/load-progress")
+ progress = progress_body if isinstance(progress_body, dict) else {}
+ if progress.get("phase") == "error":
+ break
+ time.sleep(5.0)
+ detail["load_status_body"] = {
+ k: status.get(k) for k in ("loaded", "repo_id", "family", "device", "model_kind")
+ }
+ detail["load_progress"] = {k: progress.get(k) for k in ("phase", "fraction", "error")}
+ if not status.get("loaded"):
+ failures.append(
+ f"the diffusion model never reported loaded within "
+ f"{self.args.export_deadline}s: status={detail['load_status_body']} "
+ f"progress={detail['load_progress']}"
+ )
+ detail["failures"] = failures
+ return self.record("image_generation", False, detail)
+
+ code, body = self.studio.post(
+ "/api/inference/images/generate",
+ {
+ "prompt": "a red square on a blue background",
+ "width": want,
+ "height": want,
+ "steps": 2,
+ "guidance": 0.0,
+ "seed": 3407,
+ },
+ timeout = self.args.export_timeout,
+ )
+ detail["generate_status"] = code
+ images = (body or {}).get("images") if isinstance(body, dict) else None
+ detail["generated_count"] = len(images or [])
+ if code >= 400 or not images:
+ failures.append(
+ f"images/generate returned HTTP {code} with {len(images or [])} "
+ f"images: {str(body)[:300]}"
+ )
+ detail["failures"] = failures
+ return self.record("image_generation", False, detail)
+
+ record = images[0]
+ detail["record"] = {k: record.get(k) for k in ("id", "width", "height", "steps")}
+ image_id = record.get("id")
+
+ # The PNG itself, fetched raw. The JSON client decodes to utf-8,
+ # which would corrupt the bytes the whole check is about.
+ request = urllib.request.Request(
+ f"http://127.0.0.1:{self.args.port}/api/inference/images/gallery/{image_id}/file"
+ )
+ if self.studio.token:
+ request.add_header("Authorization", f"Bearer {self.studio.token}")
+ with urllib.request.urlopen(request, timeout = 120) as response:
+ png = response.read()
+ detail["content_type"] = response.headers.get("Content-Type")
+ detail["png_bytes"] = len(png)
+
+ if not png.startswith(b"\x89PNG\r\n\x1a\n"):
+ failures.append(
+ f"the download did not start with the PNG magic, so what it "
+ f"served is not a PNG: {png[:16]!r}"
+ )
+ else:
+ # IHDR is fixed-offset: 8 magic, 4 length, 4 type, then width
+ # and height as big-endian uint32.
+ width = int.from_bytes(png[16:20], "big")
+ height = int.from_bytes(png[20:24], "big")
+ detail["png_size"] = [width, height]
+ if (width, height) != (want, want):
+ failures.append(
+ f"the PNG header says {width}x{height}, not {want}x{want} -- "
+ f"the file is not what was asked for, whatever the record says"
+ )
+
+ flat = None
+ try:
+ from PIL import Image # noqa: PLC0415
+
+ with Image.open(io.BytesIO(png)) as opened:
+ bands = opened.convert("RGB").getextrema()
+ detail["extrema"] = bands
+ flat = all(low == high for low, high in bands)
+ detail["flatness_source"] = "pixels"
+ except Exception as exc: # noqa: BLE001
+ detail["pil_error"] = f"{type(exc).__name__}: {exc}"[:200]
+ # A uniform 256x256 frame compresses to well under 2 KB.
+ flat = len(png) < 2000
+ detail["flatness_source"] = "compressed size"
+ if flat:
+ failures.append(
+ f"the image is one flat colour ({detail['flatness_source']}), "
+ f"which is what a pipeline whose weights never loaded "
+ f"produces -- and it is a perfectly valid PNG"
+ )
+ except BaseException as exc: # noqa: BLE001
+ failures.append(f"the image path raised: {type(exc).__name__}: {exc}"[:300])
+ finally:
+ # Always, or a diffusion pipeline holds the card for whatever runs
+ # next and that failure lands on the wrong assertion.
+ try:
+ self.studio.post("/api/inference/images/unload", {}, timeout = 120)
+ except BaseException: # noqa: BLE001
+ pass
+
+ detail["failures"] = failures
+ return self.record("image_generation", not failures, detail)
+
+ def assert_cloudflare(self) -> bool:
+ """`unsloth run --cloudflare`: a public URL that serves, and refuses.
+
+ This is the only assertion in the payload that reaches the public
+ internet, so what it claims is deliberately narrow and what it refuses
+ to claim is stated:
+
+ 1. cloudflared is fetched and a quick tunnel is established, and the
+ URL printed is a real `*.trycloudflare.com` host rather than the
+ `api.trycloudflare.com` that appears in cloudflared's own FAILURE
+ lines -- that is a live trap, and Studio's own regex carries the
+ same negative lookahead for it;
+ 2. the tunnel SERVES: `/api/health` answers through the public URL,
+ which is what separates "a URL was printed" from "a URL that works";
+ 3. the tunnel REFUSES an unauthenticated request. A public URL onto a
+ CI machine is only defensible if it is behind auth, and this is the
+ check that says so rather than assuming it.
+
+ `--host 0.0.0.0` is not incidental. Studio raises a quick tunnel for
+ WILDCARD binds; on 127.0.0.1 there is nothing to publish and no URL is
+ printed, which would read as a broken feature.
+
+ A tunnel that cannot be established AT ALL is reported rather than
+ failed, and the reason is carried from the log. Kaggle egress to
+ cloudflared's release host is not something this repo controls, and the
+ directive asks for this "if possible". The narrowness is enforced: the
+ excuse applies only when NO url was printed, and never once one was.
+ """
+ failures: list[str] = []
+ detail: dict = {}
+ port = self.args.port + 2
+ log_path = self.outdir / "unsloth_cloudflare.log"
+ detail["port"] = port
+
+ head = self.studio_command()[:1] or [sys.executable]
+ if head[0] == sys.executable:
+ head = [sys.executable, "-c", "from unsloth_cli import app; app()"]
+ cmd = head + [
+ "run",
+ "--model",
+ self.args.chat_model,
+ "--port",
+ str(port),
+ # WILDCARD. A loopback bind publishes nothing and prints no URL.
+ "--host",
+ "0.0.0.0",
+ "--api-only",
+ "--cloudflare",
+ "--start-api-key-marker",
+ "--max-seq-length",
+ str(self.args.studio_ctx),
+ ]
+ if self.args.chat_variant:
+ cmd += ["--gguf-variant", self.args.chat_variant]
+ detail["command"] = " ".join(cmd)
+
+ env = dict(os.environ)
+ env["UNSLOTH_STUDIO_HOME"] = str(self.studio_home)
+ env.setdefault("HF_HOME", str(self.studio_home / "cache" / "huggingface"))
+ env["PYTHONUNBUFFERED"] = "1"
+ env["UNSLOTH_DISABLE_STATISTICS"] = "1"
+
+ handle = open(log_path, "ab")
+ proc = subprocess.Popen(
+ cmd,
+ cwd = str(self.repo_root),
+ env = env,
+ stdout = handle,
+ stderr = subprocess.STDOUT,
+ )
+
+ api_key = None
+ url = None
+ try:
+ deadline = time.time() + self.args.health_deadline
+ while time.time() < deadline:
+ if proc.poll() is not None:
+ failures.append(
+ f"`unsloth run --cloudflare` exited with code {proc.returncode}"
+ )
+ break
+ text = log_path.read_text(encoding = "utf-8", errors = "replace")
+ if api_key is None and "UNSLOTH_START_API_KEY:" in text:
+ api_key = text.split("UNSLOTH_START_API_KEY:", 1)[1].split("\n", 1)[0].strip()
+ if api_key:
+ self.secrets.add(api_key)
+ if url is None:
+ # The SAME negative lookahead Studio's own matcher uses.
+ # `api.trycloudflare.com` appears in cloudflared's failure
+ # lines and is never a usable tunnel.
+ found = re.search(r"https://(?!api\.)[A-Za-z0-9-]+\.trycloudflare\.com", text)
+ if found:
+ url = found.group(0)
+ if url and api_key:
+ break
+ time.sleep(2.0)
+
+ detail["tunnel_url_seen"] = bool(url)
+ if url is None:
+ # Reported, not failed, and ONLY here: nothing was published,
+ # so there is nothing to have gone wrong with. The reason comes
+ # off the log rather than being assumed.
+ tail = log_path.read_text(encoding = "utf-8", errors = "replace")[-600:]
+ detail["no_tunnel_reason"] = self.scrub(tail)
+ detail["reported_not_failed"] = True
+ else:
+ # The host is a secret in the sense that matters here: it is a
+ # live public route to this machine, and the artifact is read
+ # by people who are not running it.
+ self.secrets.add(url)
+ public = Studio(url, timeout = 30.0)
+ code, body = public.get("/api/health", auth = False)
+ detail["public_health_status"] = code
+ # A tunnel that resolves but serves Cloudflare's own error page
+ # answers 530, which is a URL that does not work.
+ if code != 200:
+ failures.append(
+ f"the quick tunnel URL answered HTTP {code} on /api/health, "
+ f"so a URL was published that does not serve"
+ )
+
+ # And it must REFUSE. A public URL onto a CI box is only
+ # defensible behind auth, so this is asserted, not assumed.
+ code, _ = public.post(
+ "/v1/chat/completions",
+ {
+ "model": "default",
+ "messages": [{"role": "user", "content": "hi"}],
+ "max_tokens": 8,
+ },
+ auth = False,
+ )
+ detail["public_unauthenticated_status"] = code
+ if code < 400:
+ failures.append(
+ f"an UNAUTHENTICATED request through the public tunnel "
+ f"was accepted ({code}), so the quick tunnel exposes "
+ f"this machine's inference to anyone with the URL"
+ )
+ except BaseException as exc: # noqa: BLE001
+ failures.append(f"driving the tunnel raised: {type(exc).__name__}: {exc}"[:300])
+ finally:
+ proc.terminate()
+ try:
+ proc.wait(timeout = 60)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ handle.close()
+
+ detail["failures"] = failures
+ return self.record("cloudflare", not failures, detail)
+
# ------------------------------------------------------ existing drivers
+ def assert_cli_run(self) -> bool:
+ """`unsloth run`: a model server started from the CLI, driven by its key.
+
+ This is the headless path a user scripts, and nothing in CI covers it.
+ It is a different launch from `unsloth studio`: `run` starts the
+ backend, waits for health, mints an API key IN-PROCESS, and then loads
+ the model over HTTP. Any of those four can break without the others
+ noticing, and the command still prints a banner.
+
+ Four claims, and each rules out a way the previous one passes hollow:
+
+ 1. the server becomes healthy on the port it was given;
+ 2. the model reaches the GPU -- measured as device VRAM growth across
+ the launch, not as a line in the banner. `--api-only` on a card the
+ chat-UI phase has already emptied makes that delta this launch's;
+ 3. the key the command minted AUTHENTICATES a real completion, and the
+ completion is non-empty. A key that is printed and rejected is
+ worse than no key, because the failure surfaces in a user's
+ integration rather than here;
+ 4. a CORRUPTED key is refused. Without it, a server ignoring the header
+ entirely passes claim 3.
+
+ `--start-api-key-marker` is how the key is obtained: it prints
+ `UNSLOTH_START_API_KEY: <key>`, which is the mechanism `unsloth start`
+ itself uses. The value is registered as a secret the moment it is read,
+ before anything scrubs a log on the way into the evidence bundle.
+ """
+ failures: list[str] = []
+ detail: dict = {}
+ port = self.args.port + 1
+ log_path = self.outdir / "unsloth_run.log"
+ detail["port"] = port
+
+ baseline = nvidia_used_mib()
+ detail["vram_before_mib"] = baseline
+ # The per-process reading is taken alongside the device one because the
+ # device one is only valid when this payload owns the card, and under
+ # --studio-concurrent it does not. See the verdict below.
+ apps_before = nvidia_compute_apps() or {}
+ detail["compute_apps_before"] = apps_before
+
+ head = self.studio_command()[:1] or [sys.executable]
+ if head[0] == sys.executable:
+ head = [sys.executable, "-c", "from unsloth_cli import app; app()"]
+ cmd = head + [
+ "run",
+ "--model",
+ self.args.chat_model,
+ "--port",
+ str(port),
+ "--host",
+ "127.0.0.1",
+ # Headless: no UI to serve and no browser to open, which is the
+ # shape this path is for.
+ "--api-only",
+ # Never a public URL from CI. --secure would imply one.
+ "--no-cloudflare",
+ "--start-api-key-marker",
+ "--max-seq-length",
+ str(self.args.studio_ctx),
+ ]
+ if self.args.chat_variant:
+ cmd += ["--gguf-variant", self.args.chat_variant]
+ detail["command"] = " ".join(cmd)
+
+ env = dict(os.environ)
+ env["UNSLOTH_STUDIO_HOME"] = str(self.studio_home)
+ env.setdefault("HF_HOME", str(self.studio_home / "cache" / "huggingface"))
+ env["PYTHONUNBUFFERED"] = "1"
+ env["UNSLOTH_DISABLE_STATISTICS"] = "1"
+
+ handle = open(log_path, "ab")
+ proc = subprocess.Popen(
+ cmd,
+ cwd = str(self.repo_root),
+ env = env,
+ stdout = handle,
+ stderr = subprocess.STDOUT,
+ )
+
+ api_key = None
+ client = Studio(f"http://127.0.0.1:{port}")
+ try:
+ deadline = time.time() + self.args.health_deadline
+ while time.time() < deadline:
+ if proc.poll() is not None:
+ failures.append(f"`unsloth run` exited early with code {proc.returncode}")
+ break
+ text = log_path.read_text(encoding = "utf-8", errors = "replace")
+ if api_key is None and "UNSLOTH_START_API_KEY:" in text:
+ api_key = text.split("UNSLOTH_START_API_KEY:", 1)[1].split("\n", 1)[0].strip()
+ if api_key:
+ # Before anything else reads this file.
+ self.secrets.add(api_key)
+ if api_key and health_is_ready(client.get("/api/health", auth = False)[1]):
+ break
+ time.sleep(2.0)
+
+ detail["saw_api_key"] = bool(api_key)
+ if not api_key:
+ failures.append(
+ "`unsloth run` never printed UNSLOTH_START_API_KEY, so it "
+ "did not reach the point where it mints a key"
+ )
+
+ if api_key:
+ client.token = api_key
+ code, body = client.post(
+ "/v1/chat/completions",
+ {
+ "model": "default",
+ "messages": [{"role": "user", "content": "Say hello in one word."}],
+ "max_tokens": 32,
+ "temperature": 0.0,
+ },
+ timeout = self.args.chat_timeout,
+ )
+ detail["completion_status"] = code
+ text = ""
+ if code == 200 and isinstance(body, dict):
+ text = ((body.get("choices") or [{}])[0].get("message") or {}).get(
+ "content"
+ ) or ""
+ detail["generated"] = text[:200]
+ if code != 200:
+ failures.append(
+ f"the key `unsloth run` minted did not authenticate a "
+ f"completion: HTTP {code} {str(body)[:200]}"
+ )
+ elif not text.strip():
+ failures.append("the CLI-served model returned empty content")
+
+ # And a corrupted key must be refused, or the check above
+ # passes on a server that ignores the header.
+ client.token = api_key[:-4] + "0000" if len(api_key) > 8 else "bogus"
+ code, _ = client.post(
+ "/v1/chat/completions",
+ {
+ "model": "default",
+ "messages": [{"role": "user", "content": "hi"}],
+ "max_tokens": 8,
+ },
+ timeout = self.args.chat_timeout,
+ )
+ detail["bad_key_status"] = code
+ if code < 400:
+ failures.append(
+ f"a corrupted API key was accepted ({code}), so the "
+ f"check above passes whatever is sent"
+ )
+
+ # AFTER a completion has come back, not after the API-key banner.
+ # `unsloth run` prints the key while it is still starting, and on
+ # kernel unsloth-probe-studio-full2-815a0c the sample landed before
+ # llama-server had the weights anywhere: 0.0 MiB of growth on a
+ # launch whose own log says
+ # `Starting llama-server: ... -ngl -1 --fit off`, which is Studio
+ # asking for every layer on the card. A served completion is the
+ # only cheap proof the weights are resident, so the ruler goes
+ # after it.
+ settled = nvidia_used_mib()
+ detail["vram_after_mib"] = settled
+ apps_after = nvidia_compute_apps()
+ detail["compute_apps"] = apps_after
+ failure, measured = cli_run_gpu_failure(
+ apps_before,
+ apps_after,
+ baseline,
+ settled,
+ )
+ detail.update(measured)
+ if failure:
+ failures.append(failure)
+ except BaseException as exc: # noqa: BLE001
+ failures.append(f"driving `unsloth run` raised: {type(exc).__name__}: {exc}"[:300])
+ finally:
+ proc.terminate()
+ try:
+ proc.wait(timeout = 60)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ handle.close()
+
+ detail["failures"] = failures
+ return self.record("cli_run", not failures, detail)
+
def assert_chat_ui(self) -> bool:
"""Drive the repo's own chat UI driver. Runs last: it stops the server."""
driver = self.repo_root / "tests" / "studio" / "playwright_chat_ui.py"
@@ -1189,7 +2588,16 @@ class Payload:
def _pack(*, with_screenshots: bool, log_tail_bytes: int | None = None) -> bytes:
buf = io.BytesIO()
with tarfile.open(fileobj = buf, mode = "w:gz") as tar:
- for name in ("studio_gpu_report.json", "playwright_chat_ui.log", "studio.log"):
+ for name in (
+ "studio_gpu_report.json",
+ "unsloth_cloudflare.log",
+ "playwright_chat_ui.log",
+ "studio.log",
+ # `unsloth run` prints the API key it mints; redacted() is
+ # what keeps it out of the artifact, and a file nobody
+ # packs is a file nobody redacts either.
+ "unsloth_run.log",
+ ):
path = self.outdir / name
if path.is_file():
scrubbed = self.redacted(path)
@@ -1246,9 +2654,21 @@ class Payload:
if not self.authenticate():
return self.finish()
+ # Before the GPU work: it needs nothing but a logged-in session, and
+ # putting it after a 20-minute training run would mean a training
+ # failure hides whether API keys work at all.
+ self.assert_api_key()
+
+ # Before any model work: these are pure route checks that need only a
+ # logged-in session, and putting them after a 20-minute training run
+ # would mean a training failure hides whether the tabs exist at all.
+ self.assert_tabs()
+
gpu_ok = self.assert_gpu_inference()
if gpu_ok:
self.assert_tool_calling()
+ self.assert_code_execution()
+ self.assert_web_search()
else:
# Tool calling on a CPU fallback would be a green tick for a
# question nobody asked. Skip it and say so.
@@ -1257,6 +2677,48 @@ class Payload:
False,
{"failures": ["skipped: the model was not on the GPU, so this proves nothing"]},
)
+ self.record(
+ "code_execution",
+ False,
+ {"failures": ["skipped: the model was not on the GPU, so this proves nothing"]},
+ )
+ self.record(
+ "web_search",
+ False,
+ {"failures": ["skipped: the model was not on the GPU, so this proves nothing"]},
+ )
+
+ # AFTER the GPU inference assertions and BEFORE training: it reloads
+ # the chat model with different flags, so running it earlier would
+ # change the model the inference checks measured, and running it after
+ # training would put a reload between the adapter and the export.
+ if gpu_ok:
+ self.assert_server_flags()
+ # AFTER the reload, because that is what pins the window to
+ # --studio-ctx: a compaction check against an unknown context
+ # length cannot say whether the prompt was over it.
+ self.assert_compaction()
+ else:
+ self.record(
+ "compaction",
+ False,
+ {
+ "failures": [
+ "skipped: the model was not on the GPU, so the window "
+ "the check overflows was never pinned"
+ ]
+ },
+ )
+ self.record(
+ "server_flags",
+ False,
+ {
+ "failures": [
+ "skipped: the model was not on the GPU, so a "
+ "tensor_split check proves nothing"
+ ]
+ },
+ )
trained = self.assert_training()
adapter_dir = None
@@ -1265,8 +2727,47 @@ class Payload:
adapter_dir = entry.get("output_dir")
self.assert_gguf_export(adapter_dir if trained else None)
+ # Straight after, and it reads the export's own recorded path: the
+ # comparison is only meaningful against the file that assertion made.
+ exported_gguf = None
+ for entry in self.assertions:
+ if entry["name"] == "gguf_export":
+ exported_gguf = entry.get("gguf")
+ self.assert_lora_vs_base(exported_gguf)
+
+ # BEFORE the UI driver, because assert_chat_ui ends by stopping the
+ # server and every request below it would then be refused at the
+ # socket. Measured on kernel unsloth-probe-studio-full2-815a0c, where
+ # this assertion reported `URLError: Connection refused` and read as a
+ # broken image path on a server that had simply been shut down.
+ # Placed after all the language work regardless: a diffusion pipeline
+ # is the largest single thing this payload puts on a T4, and it is
+ # unloaded in a finally either way.
+ if self.args.image_generation:
+ self.assert_image_generation()
+
if not self.args.skip_ui:
self.assert_chat_ui()
+
+ # LAST, and the order is the design. `unsloth run` starts a SECOND
+ # backend against the same studio home, and two backends sharing one
+ # home's state is not a configuration anybody ships. assert_chat_ui
+ # ends by stopping the server, so by here the port is free, the card is
+ # empty, and the VRAM delta below measures this launch alone.
+ self.assert_cli_run()
+
+ # LAST of all, and the only thing here that touches the public
+ # internet. Its own launch rather than a flag on the one above,
+ # because the tunnel needs a WILDCARD bind and assert_cli_run's claim
+ # is about a loopback server.
+ if self.args.cloudflare_check:
+ self.assert_cloudflare()
+ else:
+ self.record(
+ "cloudflare",
+ False,
+ {"failures": ["skipped: --no-cloudflare-check was passed"]},
+ )
return self.finish()
def finish(self) -> int:
@@ -1298,9 +2799,54 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
ap.add_argument("--repo-root", required = True, help = "the unsloth checkout under test")
ap.add_argument("--studio-home", required = True, help = "UNSLOTH_STUDIO_HOME for this run")
ap.add_argument("--port", type = int, default = 18902)
- ap.add_argument("--chat-model", default = "unsloth/Qwen3.5-2B-GGUF")
+ ap.add_argument(
+ # 2048 constrains the runs, as the brief asks. It also keeps the
+ # KV-cache check about the cache: an unconstrained context on a 14.56GB
+ # card turns it into an OOM about something else.
+ "--studio-ctx",
+ dest = "studio_ctx",
+ type = int,
+ default = 2048,
+ help = "context length to pin the llama.cpp server to",
+ )
+ ap.add_argument(
+ "--image-model",
+ dest = "image_model",
+ default = "unsloth/sdxl-turbo",
+ help = "diffusion repo for the image-generation assertion",
+ )
+ ap.add_argument(
+ # OFF by default: it is the last-priority item and it pulls a diffusion
+ # checkpoint the rest of the payload has no use for. A dispatch that
+ # wants it says so.
+ "--image-generation",
+ dest = "image_generation",
+ action = "store_true",
+ default = False,
+ help = "load a diffusion model and generate one 256x256 image at 2 steps",
+ )
+ ap.add_argument(
+ # On by default because the directive asks for it, and off-able because
+ # it is the one assertion here that reaches the public internet.
+ "--no-cloudflare-check",
+ dest = "cloudflare_check",
+ action = "store_false",
+ default = True,
+ help = "skip the quick-tunnel assertion (it opens a public URL)",
+ )
+ ap.add_argument(
+ # Empty means "use the bootstrap password", which is the behaviour this
+ # payload had before. A caller passes `auto` to have one generated.
+ "--studio-password",
+ default = "",
+ help = "start Studio with --password and log in with it; 'auto' generates one",
+ )
+ # MTP-GGUF, not the plain GGUF: multi-token prediction is a distinct
+ # serving path in llama.cpp, and a leg pointed at the plain repo cannot
+ # tell whether it works.
+ ap.add_argument("--chat-model", default = "unsloth/Qwen3.5-2B-MTP-GGUF")
ap.add_argument("--chat-variant", default = "UD-Q4_K_XL")
- ap.add_argument("--train-model", default = "unsloth/Qwen2.5-0.5B-Instruct")
+ ap.add_argument("--train-model", default = "unsloth/Qwen3.5-2B")
ap.add_argument("--max-steps", type = int, default = 8)
ap.add_argument("--quantization", default = "q8_0")
ap.add_argument(
diff --git a/tests/kaggle/studio_gpu/studio_client.py b/tests/kaggle/studio_gpu/studio_client.py
index daf5b019952..6320a2ca273 100644
--- a/tests/kaggle/studio_gpu/studio_client.py
+++ b/tests/kaggle/studio_gpu/studio_client.py
@@ -368,9 +368,22 @@ def nonfinite_losses(status: Any) -> list:
def newest_gguf(root: str | Path) -> Path | None:
- """The most recently written ``.gguf`` under ``root``, if any."""
+ """The most recently written MODEL ``.gguf`` under ``root``, if any.
+
+ ``mmproj`` sidecars are excluded, and that is not a tidy-up. A vision export
+ writes two files -- ``Qwen3.5-2B.Q8_0.gguf`` and
+ ``Qwen3.5-2B.F16-mmproj.gguf`` -- and the projector is often the newer of
+ the two. Handing it to llama.cpp as a model is not an error: the server
+ starts, reports ``gpu_layers=-1``, offloads nothing and still returns text,
+ so the run reads as a GPU failure in the export assertion. Measured on
+ kernel unsloth-probe-studio-full2-815a0c, where exactly that happened.
+ """
root = Path(root)
if not root.is_dir():
return None
- candidates = sorted(root.rglob("*.gguf"), key = lambda p: p.stat().st_mtime, reverse = True)
+ candidates = sorted(
+ (p for p in root.rglob("*.gguf") if "mmproj" not in p.name.lower()),
+ key = lambda p: p.stat().st_mtime,
+ reverse = True,
+ )
return candidates[0] if candidates else None
diff --git a/tests/kaggle/t4_smoke/gguf_export.py b/tests/kaggle/t4_smoke/gguf_export.py
new file mode 100644
index 00000000000..1c57e884bb4
--- /dev/null
+++ b/tests/kaggle/t4_smoke/gguf_export.py
@@ -0,0 +1,279 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""Export a trained model to GGUF, and prove the file exists and is usable.
+
+Shared by the payloads, in the shape the rest of this directory uses: nothing
+here raises, everything returns a record, and a separate pure function turns
+that record into failures. A diagnostic that kills the payload it diagnoses
+leaves the leg reporting nothing at all.
+
+Every constant below is a measurement, not a guess. The two that a natural
+implementation gets WRONG, both learned by getting them wrong:
+
+**1. The GGUF is not in the directory you passed.** On kernel
+unsloth-probe-gguf-q8-peft-920e3e, `save_pretrained_gguf("/tmp/q8p", ...)` put
+the merged `model.safetensors` (1136.9 MB) in `/tmp/q8p` and the actual
+`qwen3-0.6b.Q8_0.gguf` (609.8 MB) in the SIBLING `/tmp/q8p_gguf`. Code that
+globs the directory it passed finds no `.gguf`, raises nothing, and reports a
+successful export. So this searches both and records which one held the file.
+
+**2. `install_llama_cpp()` returns a TUPLE**, `(llama-quantize,
+convert_hf_to_gguf.py)`, not a path to a bin directory. An earlier probe
+treated it as a directory, found nothing in it, and reported "0 binaries" --
+which was the probe being wrong rather than the bundle being empty.
+
+Two more facts worth having here rather than rediscovering:
+
+* llama.cpp installs PREBUILT on Kaggle, about 10-46s, logging "Installing
+ prebuilt llama.cpp ... - skipping compilation". A source build is silent,
+ correct, and costs many minutes on 4 vCPUs, so "it worked" is not the
+ interesting assertion -- "it did not compile" is.
+* The bundle is the `-cpu` one, so GGUF inference runs on CPU. Fine for a 0.6B
+ or a 2B; a reason not to lean on GGUF inference for a 20B.
+
+A model that REFUSES the requested quantization is not a failure. gpt-oss
+answers q8_0 with "GPT-OSS does not support GGUF quantization (requested:
+q8_0). Overriding to MXFP4 format", by design, so the caller says what it will
+accept rather than this module assuming q8_0 is universal.
+"""
+
+from __future__ import annotations
+
+import os
+import subprocess
+import time
+
+# Where a GGUF may legitimately appear, relative to the directory passed to
+# save_pretrained_gguf. The empty string is that directory itself; "_gguf" is
+# the sibling unsloth actually writes to. Both are searched because relying on
+# either alone has already produced a wrong answer.
+GGUF_SEARCH_SUFFIXES = ("", "_gguf")
+
+# Marks the prebuilt branch in unsloth's own install output.
+PREBUILT_MARKER = "skipping compilation"
+
+# Substrings that mean a source build happened instead. If any appears, the
+# leg took the expensive path and should say so even though it succeeded.
+SOURCE_BUILD_MARKERS = ("cmake", "Building llama.cpp", "make -j")
+
+
+def find_ggufs(save_dir: str) -> list:
+ """Every .gguf reachable from `save_dir`, with the directory that held it."""
+ found = []
+ for suffix in GGUF_SEARCH_SUFFIXES:
+ candidate = save_dir + suffix
+ if not os.path.isdir(candidate):
+ continue
+ for root, _dirs, files in os.walk(candidate):
+ for name in files:
+ if not name.endswith(".gguf"):
+ continue
+ path = os.path.join(root, name)
+ found.append(
+ {
+ "path": path,
+ "mb": round(os.path.getsize(path) / 1024**2, 1),
+ "found_in": candidate,
+ "suffix": suffix,
+ }
+ )
+ return sorted(found, key = lambda f: -f["mb"])
+
+
+def export_gguf(
+ model,
+ tokenizer,
+ save_dir: str,
+ *,
+ quantization: str = "q8_0",
+) -> dict:
+ """Export and report. Never raises."""
+ record = {"save_dir": save_dir, "requested_quantization": quantization}
+
+ started = time.time()
+ try:
+ model.save_pretrained_gguf(save_dir, tokenizer, quantization_method = quantization)
+ record["ok"] = True
+ except BaseException as exc: # noqa: BLE001
+ # unsloth/save.py:4777 wraps the real cause in
+ # `RuntimeError(f"Failed to save model: {e}")` and interpolates str(e)
+ # from exceptions whose args are empty, producing a message that ends
+ # at the colon with no cause in it. Record the type too, so a report
+ # from that path still names something.
+ record["ok"] = False
+ record["error"] = f"{type(exc).__name__}: {exc}"[:4000]
+ record["seconds"] = round(time.time() - started, 1)
+
+ record["ggufs"] = find_ggufs(save_dir)
+ return record
+
+
+def run_gguf(
+ gguf_path: str,
+ llama_cpp_dir: str,
+ *,
+ max_tokens: int = 16,
+ timeout: int = 240,
+) -> dict:
+ """Run the exported file, because an existing GGUF is not a working one.
+
+ NOT via `llama-cli`. It hung for its entire budget twice on Kaggle -- 600s
+ on kernel unsloth-probe-gguf-q8-peft-920e3e and 180s on -gguf-infer-50a98a,
+ the second with stdin closed AND `-no-cnv` -- while 16 tokens from a 610 MB
+ Q8_0 on CPU is seconds of work. Whatever it waits for is not stdin, so this
+ uses binaries that cannot be interactive at all: `llama-bench` runs a fixed
+ workload and exits, `llama-completion` is one-shot.
+ """
+ record = {"gguf": gguf_path}
+ for name, argv in (
+ ("bench", ["llama-bench", "-m", gguf_path, "-p", "8", "-n", str(max_tokens), "-r", "1"]),
+ (
+ "completion",
+ [
+ "llama-completion",
+ "-m",
+ gguf_path,
+ "-p",
+ "The capital of France is",
+ "-n",
+ str(max_tokens),
+ "--temp",
+ "0",
+ ],
+ ),
+ ):
+ exe = os.path.join(llama_cpp_dir, argv[0])
+ if not os.path.exists(exe):
+ record[name] = {"skipped": f"no {argv[0]} in the bundle"}
+ continue
+ started = time.time()
+ try:
+ proc = subprocess.run(
+ [exe] + argv[1:],
+ capture_output = True,
+ text = True,
+ timeout = timeout,
+ stdin = subprocess.DEVNULL,
+ )
+ record[name] = {
+ "seconds": round(time.time() - started, 1),
+ "returncode": proc.returncode,
+ "stdout": proc.stdout[-4000:],
+ "stderr": proc.stderr[-2000:],
+ }
+ except BaseException as exc: # noqa: BLE001
+ record[name] = {
+ "seconds": round(time.time() - started, 1),
+ "error": f"{type(exc).__name__}: {exc}"[:1000],
+ }
+ # One success is enough; the second is only run when the first is
+ # unavailable or failed, so a healthy leg pays for one process.
+ if record[name].get("returncode") == 0:
+ break
+ return record
+
+
+def export_failures(record: dict, *, accept_quantizations = None) -> list:
+ """The pass rule, as a pure function of the record.
+
+ `accept_quantizations` is the set of quantization names the caller will
+ accept in the filename. Callers pass more than one where the model is
+ allowed to override the request -- gpt-oss answers q8_0 with MXFP4 by
+ design, and failing on that would be failing on documented behaviour.
+ """
+ if not record:
+ return ["GGUF export was never run"]
+
+ failures = []
+ if not record.get("ok"):
+ failures.append("GGUF export raised: " + str(record.get("error", "no error recorded")))
+
+ ggufs = record.get("ggufs") or []
+ if not ggufs:
+ # The trap, stated in the failure itself so the reader does not have to
+ # know it: an export can "succeed" and leave no GGUF anywhere.
+ failures.append(
+ f"no .gguf under {record.get('save_dir')!r} or its _gguf sibling, "
+ f"even though the export reported ok={record.get('ok')}"
+ )
+ return failures
+
+ biggest = ggufs[0]
+ if biggest["mb"] <= 1.0:
+ failures.append(
+ f"the largest .gguf is {biggest['mb']} MB ({biggest['path']}), which is "
+ f"a header and no weights"
+ )
+
+ if accept_quantizations:
+ names = [os.path.basename(g["path"]).lower() for g in ggufs]
+ accepted = [q.lower() for q in accept_quantizations]
+ if not any(q in n for n in names for q in accepted):
+ failures.append(
+ f"no exported file names any accepted quantization {sorted(accept_quantizations)}: "
+ f"got {names}"
+ )
+ return failures
+
+
+def run_failures(record: dict) -> list:
+ """Did the exported file actually produce output?"""
+ if not record:
+ return ["the exported GGUF was never run"]
+
+ attempts = {k: v for k, v in record.items() if isinstance(v, dict)}
+ if not attempts:
+ return ["no runner was attempted against the exported GGUF"]
+ if all(a.get("skipped") for a in attempts.values()):
+ return [
+ "every GGUF runner was missing from the llama.cpp bundle: "
+ + ", ".join(sorted(attempts))
+ ]
+ if any(a.get("returncode") == 0 for a in attempts.values()):
+ return []
+
+ detail = "; ".join(
+ f"{name}: "
+ + (
+ a.get("error")
+ or a.get("skipped")
+ or f"rc={a.get('returncode')} {(a.get('stderr') or '')[-200:]}"
+ )
+ for name, a in sorted(attempts.items())
+ )
+ return [f"the exported GGUF produced no output from any runner ({detail})"]
+
+
+def llama_cpp_facts(install_output: str, returned) -> dict:
+ """Was the PREBUILT branch taken, and what did the installer hand back?
+
+ `returned` is whatever `install_llama_cpp()` returned -- a TUPLE of
+ (llama-quantize, convert_hf_to_gguf.py) as measured, not a directory. The
+ directory is derived from it rather than assumed, so a release that changes
+ the layout shows up as a missing binary instead of a wrong path.
+ """
+ paths = list(returned) if isinstance(returned, (tuple, list)) else [returned]
+ paths = [str(p) for p in paths]
+ return {
+ "returned": paths,
+ "all_exist": all(os.path.exists(p) for p in paths) if paths else False,
+ "dir": os.path.dirname(paths[0]) if paths else None,
+ # TRI-STATE, and the third state is the point. The prebuilt banner is
+ # printed once, by the install that actually downloads the bundle; a
+ # second cycle in the same session finds llama.cpp already there and
+ # prints nothing. Measured on unsloth-probe-visleg-full-b3a317, where
+ # cycle 0 read `prebuilt: true` and cycle 1 read `prebuilt: false` for
+ # the SAME installation -- and `false` reads as "built from source",
+ # which is the one thing this field exists to catch.
+ #
+ # None means "this run did not install it, so it cannot say". The
+ # source-build markers stay a plain list, because those are printed by
+ # the build itself and their absence is meaningful either way.
+ "prebuilt": (
+ True
+ if PREBUILT_MARKER in (install_output or "")
+ else (None if not (install_output or "").strip() else False)
+ ),
+ "source_build_markers": [m for m in SOURCE_BUILD_MARKERS if m in (install_output or "")],
+ }
diff --git a/tests/kaggle/t4_smoke/kernel_provenance.py b/tests/kaggle/t4_smoke/kernel_provenance.py
new file mode 100644
index 00000000000..2e71b7646da
--- /dev/null
+++ b/tests/kaggle/t4_smoke/kernel_provenance.py
@@ -0,0 +1,163 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""Which fast kernels a model actually got, and where each one came from.
+
+**The failure this is built to catch is not "kernel missing".** It is "kernel
+built from source", which is silent, correct, slow, and reports the same
+`__version__` as the wheel. Only the module's `__file__` and the install log
+tell them apart, so a version check here would pass while proving nothing.
+
+**Read AFTER the model load, not before.** Measured on kernel
+`unsloth-probe-vision-recon-c76ea3`: `fla` is not importable before
+`from_pretrained` and IS importable after, because unsloth reaches for it lazily
+when it sees the model. A probe that read provenance only up front reported it
+absent, which is the opposite of the truth.
+
+Nothing here raises. A payload that dies collecting a diagnostic reports nothing
+at all, which is the one outcome worse than a missing field.
+"""
+
+from __future__ import annotations
+
+# The kernels worth asking about for a Qwen3.5-class model, and what each answer
+# is allowed to mean. `_EXPECTED` is deliberately NOT "all of these must be
+# present": two of them are measured absent on this path and asserting them
+# would be red on correct behaviour. See vision_kernel_failures below.
+_KERNELS = ("fla", "causal_conv1d", "mamba_ssm", "flash_attn", "triton", "xformers")
+
+
+def probe_kernels() -> dict:
+ """Import each kernel and record where it resolved from."""
+ out: dict = {}
+ for name in _KERNELS:
+ entry: dict = {"importable": False}
+ try:
+ module = __import__(name)
+ entry["importable"] = True
+ entry["file"] = getattr(module, "__file__", None)
+ entry["version"] = getattr(module, "__version__", None)
+ # The distinction the whole module exists for. A vendored copy lives
+ # inside unsloth_zoo; a pip-installed one does not.
+ entry["vendored"] = "_vendored" in (entry["file"] or "")
+ except BaseException as exc: # noqa: BLE001
+ entry["error"] = f"{type(exc).__name__}: {exc}"[:200]
+ out[name] = entry
+
+ # Distributions separately, because a package can be installed and NOT
+ # importable -- a wheel with the wrong CUDA ABI is exactly that -- and
+ # reporting only the import would call that "absent".
+ try:
+ from importlib import metadata
+
+ dists = {}
+ for dist in metadata.distributions():
+ name = (dist.metadata["Name"] or "").lower()
+ if name in (
+ "causal-conv1d",
+ "mamba-ssm",
+ "flash-attn",
+ "fla-core",
+ "flash-linear-attention",
+ "xformers",
+ ):
+ dists[name] = dist.version
+ out["_distributions"] = dists
+ except Exception as exc: # noqa: BLE001
+ out["_distributions"] = {"error": str(exc)[:200]}
+ return out
+
+
+def attention_choice(model) -> dict:
+ """What attention resolved to, read off the config.
+
+ The config records the choice; a module walk was tried on the recon probe
+ and returned an empty set, so this reports the one source that answers.
+ """
+ record: dict = {}
+ try:
+ config = getattr(model, "config", None)
+ record["config"] = getattr(config, "_attn_implementation", None)
+ text = getattr(config, "text_config", None)
+ if text is not None:
+ record["text_config"] = getattr(text, "_attn_implementation", None)
+ except BaseException as exc: # noqa: BLE001
+ record["error"] = f"{type(exc).__name__}: {exc}"[:200]
+ return record
+
+
+def _is_turing(capability) -> bool:
+ """True for compute capability 7.x, whichever way it was spelled."""
+ text = str(capability or "").strip().lower().replace("sm_", "").replace("sm", "")
+ if not text:
+ return False
+ # "7.5" -> 7, "75" -> 7. Both spellings appear in this repo.
+ head = text.split(".")[0]
+ if "." in text:
+ return head == "7"
+ return len(head) >= 2 and head[0] == "7"
+
+
+def vision_kernel_failures(
+ kernels: dict | None, attention: dict | None, *, capability: str
+) -> list:
+ """The pass rule, as a pure function so it is checkable without a GPU.
+
+ Three claims, each chosen so it is neither false nor vacuous on a T4:
+
+ 1. **FLA is present and VENDORED.** Measured: it resolves to
+ `unsloth_zoo/_vendored/fla`, version 0.5.1, after the load. Asserting
+ merely "importable" would pass on a pip-installed copy that is not what
+ ships, which is a different thing being tested.
+ 2. **Attention is a valid Turing choice, and it is NOT flash_attention_2.**
+ FA2 supports Ampere, Ada and Hopper. On sm_75 it cannot execute, so
+ asserting it ran would be false and asserting it was "selected" would be
+ vacuous. Asserting `sdpa` (or another real Turing path) is the claim
+ that can be both true and informative -- and it catches the regression
+ that matters, which is unsloth choosing a backend the card cannot run.
+ 3. **Nothing was built from source.** A source build is silent and costs
+ many minutes on 4 vCPUs.
+
+ `causal_conv1d` and `mamba_ssm` are deliberately NOT asserted present.
+ Measured on the recon probe: neither is installed on the notebook path,
+ before or after the load. The wheel-first machinery in
+ `studio/backend/utils/ssm_runtime.py` belongs to Studio's training worker
+ and this path never calls it. Asserting them would be red on correct
+ behaviour; they are REPORTED so a change shows up in the diff.
+ """
+ if not kernels:
+ return ["no kernel provenance was collected at all"]
+
+ failures = []
+
+ fla = kernels.get("fla") or {}
+ if not fla.get("importable"):
+ failures.append(
+ f"fla did not import after the model load, so the vendored fast "
+ f"kernels are not reachable: {fla.get('error')}"
+ )
+ elif not fla.get("vendored"):
+ failures.append(
+ f"fla imported from {fla.get('file')!r}, which is not the vendored "
+ f"copy under unsloth_zoo/_vendored. This leg is about the vendored "
+ f"kernels; a pip-installed fla is a different thing"
+ )
+
+ # Normalised, because the two spellings in this repo are BOTH live and a
+ # rule that silently never fires is the exact failure this file is about:
+ # environment_fingerprint() records "sm_75", while the recon probe and
+ # torch.cuda.get_device_capability report "7.5". A startswith("7.") check
+ # against "sm_75" matches nothing, and an FA2 regression would sail past it.
+ turing = _is_turing(capability)
+ if turing:
+ chosen = (attention or {}).get("config")
+ if chosen in (None, ""):
+ failures.append("no attention implementation was recorded")
+ elif "flash_attention_2" in str(chosen):
+ failures.append(
+ f"attention resolved to {chosen!r} on capability {capability}. "
+ f"FlashAttention-2 supports Ampere, Ada and Hopper; a Turing "
+ f"card cannot run it, so this would fail at the first forward"
+ )
+
+ return failures
diff --git a/tests/kaggle/t4_smoke/measured_vram.json b/tests/kaggle/t4_smoke/measured_vram.json
new file mode 100644
index 00000000000..4257d6f6590
--- /dev/null
+++ b/tests/kaggle/t4_smoke/measured_vram.json
@@ -0,0 +1,12 @@
+{
+ "_source": "executed payload notebooks of run 32611343797 (unslothai/unsloth), Tesla T4; multi_gpu from kernel unsloth-probe-multigpu-r2-a280e2, both cycles 1.2 GB",
+ "_why": "Leg.vram_gb decides which legs may share a card and is checked against these. Recapture from a run's evidence, never by hand.",
+ "card_total_gb": 14.56,
+ "peak_reserved_gb": {
+ "canary": 0.7,
+ "control": 0.7,
+ "frontier": 0.7,
+ "gptoss": 12.78,
+ "multi_gpu": 1.2
+ }
+}
diff --git a/tests/kaggle/t4_smoke/naive_trl_compare.py b/tests/kaggle/t4_smoke/naive_trl_compare.py
new file mode 100644
index 00000000000..62467525edb
--- /dev/null
+++ b/tests/kaggle/t4_smoke/naive_trl_compare.py
@@ -0,0 +1,328 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""Train the same rows with PLAIN TRL, so an Unsloth trace has something to sit
+next to.
+
+**What this can say, and what it deliberately does not.** It does NOT assert
+that the two paths produce the same losses, and the temptation to is the whole
+reason this docstring is long. The `frontier` leg already showed transformers
+5.5.0 and 5.15.1 producing different step-1 losses on identical weights, data
+and seed (10.3222 against 6.4367), so the loss function itself moves between
+library versions. Two DIFFERENT stacks agreeing to a tolerance would be a
+coincidence, and a check built on one would go red on ordinary drift, which is
+the kind of red that gets a check switched off the week before it is right.
+
+What it asserts is the pair of claims that are version-independent: the plain
+path **runs at all** on this model and this trl, and it **converges** rather
+than sitting flat or going non-finite. Everything else is reported side by side
+for a human.
+
+**Why a separate file and a separate process.** Unsloth patches transformers,
+trl and peft at import time. Anything that has imported it is no longer a
+control, and an in-process "comparison" would be Unsloth against itself with
+extra steps. This module must never import unsloth, and the guard in
+tests/kaggle/test_t4_smoke_harness.py asserts that from the source rather than
+trusting the convention.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import random
+import sys
+import time
+from pathlib import Path
+
+_HERE = Path(__file__).resolve().parent
+
+
+def _log(msg: str) -> None:
+ print(f"[naive-trl] {msg}", flush = True)
+
+
+def _seed_everything(seed: int) -> None:
+ import numpy as np
+ import torch
+
+ random.seed(seed)
+ np.random.seed(seed)
+ torch.manual_seed(seed)
+ torch.cuda.manual_seed_all(seed)
+
+
+def _rows(path: Path) -> list[dict]:
+ out = []
+ for line in path.read_text(encoding = "utf-8").splitlines():
+ line = line.strip()
+ if line:
+ out.append(json.loads(line))
+ return out
+
+
+def run(args) -> dict:
+ """Load in 4bit, attach LoRA, train, and report the step trace."""
+ # Asserted rather than assumed: an unsloth already in sys.modules means
+ # transformers is patched and this arm is not a control. It cannot happen
+ # by accident in a fresh process, which is exactly why it would go
+ # unnoticed if it ever did.
+ if "unsloth" in sys.modules or "unsloth_zoo" in sys.modules:
+ raise RuntimeError(
+ "unsloth is imported in the plain-TRL process, so this is not a "
+ "control arm; the comparison would be unsloth against itself"
+ )
+
+ import torch
+ from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+
+ result: dict = {
+ "model": args.model,
+ "max_steps": args.max_steps,
+ "imported_unsloth": False,
+ }
+
+ _seed_everything(args.seed)
+
+ # float16 and NF4, to match what the Unsloth arm ends up on: a T4 is sm_75
+ # and has no bf16, and load_in_4bit there resolves to NF4 with a float16
+ # compute dtype. Matching the numeric path is not enough to make the losses
+ # comparable (see the module docstring) but mismatching it would add a
+ # difference nobody asked about.
+ quant = BitsAndBytesConfig(
+ load_in_4bit = True,
+ bnb_4bit_quant_type = "nf4",
+ bnb_4bit_compute_dtype = torch.float16,
+ bnb_4bit_use_double_quant = True,
+ )
+
+ t0 = time.time()
+ tokenizer = AutoTokenizer.from_pretrained(args.model)
+ model = AutoModelForCausalLM.from_pretrained(
+ args.model,
+ quantization_config = quant,
+ dtype = torch.float16,
+ device_map = {"": 0},
+ )
+ result["load_seconds"] = round(time.time() - t0, 1)
+ result["resolved_checkpoint"] = getattr(getattr(model, "config", None), "_name_or_path", None)
+
+ # The plain arm resolves its own repo, and it is NOT the one the Unsloth arm
+ # loads: Unsloth's FLOAT_TO_INT_MAPPER redirects a 16bit name to a
+ # pre-quantised `-unsloth-bnb-4bit` sibling, while this path quantises the
+ # original on the fly. Recorded rather than reconciled, because forcing them
+ # onto one repo would test a path neither a user nor the leg takes.
+
+ from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
+
+ # Gradient checkpointing ON, and leaving it off was an unfair comparison
+ # rather than a neutral one: the unsloth arm runs with
+ # `gradient_checkpointing="unsloth"`, so a control without it is measured
+ # with the single largest memory lever disabled on one side only. On
+ # gemma-4-E2B-it that is the difference between a comparison and an OOM --
+ # the control asked for 8.75GiB on top of 8.96GiB already resident, on a
+ # 14.56GiB card (kernel unsloth-probe-latestcompile-r4-e67ef2).
+ model = prepare_model_for_kbit_training(model, use_gradient_checkpointing = True)
+ model = get_peft_model(
+ model,
+ LoraConfig(
+ r = args.lora_r,
+ lora_alpha = args.lora_alpha,
+ lora_dropout = 0.0,
+ bias = "none",
+ task_type = "CAUSAL_LM",
+ target_modules = [
+ "q_proj",
+ "k_proj",
+ "v_proj",
+ "o_proj",
+ "gate_proj",
+ "up_proj",
+ "down_proj",
+ ],
+ ),
+ )
+
+ from datasets import Dataset
+ from trl import SFTConfig, SFTTrainer
+
+ rows = _rows(Path(args.dataset))
+ eos = tokenizer.eos_token or ""
+ dataset = Dataset.from_list([{"text": r["prompt"] + r["completion"] + eos} for r in rows])
+
+ outdir = Path(args.outdir)
+ outdir.mkdir(parents = True, exist_ok = True)
+
+ config = SFTConfig(
+ output_dir = str(outdir / "trainer"),
+ per_device_train_batch_size = args.batch_size,
+ gradient_accumulation_steps = args.grad_accum,
+ max_steps = args.max_steps,
+ learning_rate = args.learning_rate,
+ logging_steps = 1,
+ optim = args.optim,
+ fp16 = True,
+ bf16 = False,
+ seed = args.seed,
+ report_to = [],
+ save_strategy = "no",
+ max_length = args.max_seq_length,
+ gradient_checkpointing = True,
+ # Required with gradient checkpointing on a PEFT model: without it the
+ # inputs carry no grad and the backward finds nothing to do, which
+ # surfaces as "element 0 of tensors does not require grad" rather than
+ # as a configuration mistake.
+ gradient_checkpointing_kwargs = {"use_reentrant": False},
+ )
+ trainer = SFTTrainer(model = model, train_dataset = dataset, args = config)
+
+ t0 = time.time()
+ trainer.train()
+ result["train_seconds"] = round(time.time() - t0, 1)
+
+ metrics = []
+ for entry in trainer.state.log_history:
+ if "loss" in entry:
+ metrics.append(
+ {
+ "step": int(entry.get("step", len(metrics) + 1)),
+ "loss": entry["loss"],
+ "grad_norm": entry.get("grad_norm"),
+ }
+ )
+ result["metrics"] = metrics
+ return result
+
+
+# Substrings that identify an out-of-memory failure, whichever layer raised it.
+# torch says "CUDA out of memory", accelerate and bitsandbytes wrap it, and the
+# exception TYPE is not reliably OutOfMemoryError once it has been re-raised.
+_OOM_MARKERS = ("out of memory", "outofmemoryerror", "cuda oom")
+
+
+def _is_oom(text: str) -> bool:
+ lowered = (text or "").lower()
+ return any(marker in lowered for marker in _OOM_MARKERS)
+
+
+def comparison_failures(
+ naive: dict | None,
+ unsloth_metrics: list[dict] | None,
+ *,
+ allow_oom: bool = False,
+) -> list[str]:
+ """The two claims this comparison is entitled to make.
+
+ Kept as a pure function of two dicts so the rules are testable on CPU
+ without a GPU or a model, and separated from `run` so a guard cannot pass by
+ exercising the rule against a hand-written dict alone -- see the Default
+ leg, where seven such guards all passed while the code that produces the
+ dict raised NameError on hardware.
+ """
+ if naive is None:
+ return ["the plain-TRL arm produced no report at all"]
+ if naive.get("error"):
+ if allow_oom and _is_oom(naive["error"]) and not naive.get("metrics"):
+ # An OOM BEFORE a single step is a statement about the card, not
+ # about either training stack, and it is measured: on
+ # gemma-4-E2B-it the plain arm asks for 8.75GiB with 8.96GiB
+ # already resident on a 14.56GiB T4, and it does so at LOAD --
+ # `metrics` is absent, so no step ever ran. Gradient checkpointing
+ # does not touch that, and enabling it changed nothing.
+ #
+ # The likely cause is worth naming rather than implying: E2B is a
+ # MatFormer SUBMODEL of E4B and the checkpoint carries the larger
+ # weights, so a loader that does not extract the submodel
+ # materialises all of them.
+ #
+ # Narrow on purpose. An OOM DURING training is still a failure --
+ # that is a finding about the run, not about the card -- and this
+ # only ever applies when the caller opted in.
+ return []
+ return [f"the plain-TRL arm did not run: {naive['error']}"]
+
+ failures = []
+ metrics = naive.get("metrics") or []
+ if not metrics:
+ failures.append(
+ "the plain-TRL arm reported no steps, so it loaded and then trained "
+ "nothing; a comparison against an empty trace is not a comparison"
+ )
+ return failures
+
+ losses = [m.get("loss") for m in metrics]
+ if any(
+ not isinstance(v, (int, float)) or v != v or v in (float("inf"), float("-inf"))
+ for v in losses
+ ):
+ failures.append(f"the plain-TRL arm produced a non-finite loss: {losses}")
+ return failures
+
+ # Converged, stated as "the end is below the start" rather than as a rate.
+ # A tiny run on a tiny dataset has no business asserting a slope, but a
+ # trace that FINISHES no lower than it started did not learn, and that is
+ # the same rule the Unsloth arm is held to.
+ if losses[-1] >= losses[0]:
+ failures.append(
+ f"the plain-TRL arm did not converge: first loss {losses[0]}, last "
+ f"{losses[-1]}. Both arms are held to the same rule; this one is "
+ f"not about agreeing with unsloth"
+ )
+
+ # Deliberately NOT compared for equality. Two library stacks do not produce
+ # one fp16 trajectory, and asserting they do would be red on drift. The
+ # count is checked instead, because a plain arm that silently ran fewer
+ # steps is reported beside a full unsloth trace as though they were the
+ # same experiment.
+ if unsloth_metrics:
+ if len(metrics) != len(unsloth_metrics):
+ failures.append(
+ f"the two arms ran different numbers of steps: plain TRL "
+ f"{len(metrics)}, unsloth {len(unsloth_metrics)}. The traces are "
+ f"reported side by side and a reader would compare them"
+ )
+ return failures
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--model", required = True)
+ ap.add_argument("--dataset", default = str(_HERE / "canary_dataset.jsonl"))
+ ap.add_argument("--outdir", required = True)
+ ap.add_argument("--max-steps", type = int, default = 10)
+ ap.add_argument("--batch-size", type = int, default = 2)
+ ap.add_argument("--grad-accum", type = int, default = 1)
+ ap.add_argument("--max-seq-length", type = int, default = 512)
+ ap.add_argument("--learning-rate", type = float, default = 1e-3)
+ ap.add_argument("--lora-r", type = int, default = 16)
+ ap.add_argument("--lora-alpha", type = int, default = 32)
+ ap.add_argument("--optim", default = "adamw_8bit")
+ ap.add_argument("--seed", type = int, default = 3407)
+ args = ap.parse_args()
+
+ outdir = Path(args.outdir)
+ outdir.mkdir(parents = True, exist_ok = True)
+ report_path = outdir / "naive_trl_report.json"
+
+ # A crash here must not take the leg down: this arm is a COMPARISON, and a
+ # leg whose unsloth run passed should not go red because the control could
+ # not install. The failure is recorded, reported, and left for
+ # comparison_failures to rule on.
+ try:
+ result = run(args)
+ except BaseException as exc: # noqa: BLE001
+ text = str(exc).strip()
+ result = {
+ "model": args.model,
+ "error": f"{type(exc).__name__}: {text}" if text else type(exc).__name__,
+ }
+ _log(f"failed: {result['error']}")
+
+ report_path.write_text(json.dumps(result, indent = 2), encoding = "utf-8")
+ print("NAIVE_TRL_REPORT " + json.dumps(result), flush = True)
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tests/kaggle/t4_smoke/run_gptoss_t4.py b/tests/kaggle/t4_smoke/run_gptoss_t4.py
index 95710f3930e..f2cc86c0922 100644
--- a/tests/kaggle/t4_smoke/run_gptoss_t4.py
+++ b/tests/kaggle/t4_smoke/run_gptoss_t4.py
@@ -23,10 +23,15 @@ not in the path at all.
probe saw `UNSLOTH_FORCE_FLOAT32=1`, `fp16=False`, `bf16=False`, and
`UNSLOTH_FORCE_CUSTOM_DTYPE` pinning `down_projs` and `mlp.router` to float32.
That path exists for this card and nothing else in CI exercises it.
-* **No offload.** 12.78 GB reserved of 14.56, every parameter on `cuda:0`, no
- `hf_device_map`. About 1.8 GB of headroom, thin enough that placement is
- counted on every run rather than assumed -- and asserted, since a run that
- offloads passes everything else.
+* **One deliberate offload, and nothing else.** 12.7 GB reserved of 14.56, no
+ `hf_device_map`, every parameter on `cuda:0` EXCEPT `model.embed_tokens.weight`,
+ which unsloth puts in RAM on purpose -- `Unsloth: Offloading embeddings to RAM
+ to save 1.08 GB`, with forward hooks carrying ids down and vectors back up.
+ That was read as a spill twice before anyone looked at the name; 579133440 is
+ exactly 201088 x 2880, this checkpoint's vocab by its hidden size. Placement is
+ counted on every run rather than assumed, and the embedding is excused only
+ when its hook flag is set, because an embedding that reached the CPU without
+ them is a real bug that looks identical in a device count.
* **torch.compile engages**: 32 unique graphs, 779 calls captured, 2 graph
breaks, both `_warnings.warn`. A silent fall back to eager leaves every other
number healthy while the leg's coverage goes unexercised, so this is asserted.
@@ -72,6 +77,8 @@ import json
import os
import platform
import sys
+import shutil
+import tempfile
import time
import traceback
from pathlib import Path
@@ -152,15 +159,51 @@ def placement(model) -> dict:
landed on the CPU (correct, slow) from one on meta (loaded nothing).
"""
counts: dict = {}
+ off_gpu: list = []
try:
- for param in model.parameters():
+ for name, param in model.named_parameters():
key = str(param.device)
counts[key] = counts.get(key, 0) + param.numel()
+ if not key.startswith("cuda"):
+ # NAMED, not just counted. `{'cpu': 579133440}` is a number
+ # nobody can act on; `model.embed_tokens.weight` is a bug
+ # report. The two readings cost the same walk, and the first
+ # one already reached hardware twice before anyone could say
+ # which tensor it was.
+ off_gpu.append({"name": name, "numel": param.numel(), "device": key})
except Exception as exc: # noqa: BLE001
counts = {"error": f"{type(exc).__name__}: {exc}"}
+ # Is the one tensor allowed off the card the one unsloth DELIBERATELY put
+ # there? `offload_embedding` moves the input embedding to RAM and installs
+ # a pre/post forward hook pair that carries the ids down and the vectors
+ # back up (`unsloth/models/vision.py:_install_offload_embedding_hooks`).
+ # The flag it sets is the difference between that optimisation and a
+ # weight that landed on the CPU by accident, and the two are identical in a
+ # device count.
+ embed = {}
+ try:
+ module = model.get_input_embeddings()
+ weight = getattr(module, "weight", None)
+ for name, candidate in model.named_modules():
+ if candidate is module:
+ embed["module"] = name
+ break
+ embed["weight_name"] = f"{embed.get('module')}.weight" if "module" in embed else None
+ embed["device"] = str(weight.device) if weight is not None else None
+ embed["offload_hooks_installed"] = bool(
+ getattr(module, "_unsloth_offload_hooks_installed", False)
+ )
+ except Exception as exc: # noqa: BLE001
+ embed = {"error": f"{type(exc).__name__}: {exc}"}
+
device_map = getattr(model, "hf_device_map", None)
return {
+ "input_embedding": embed,
"parameters_by_device": counts,
+ # Largest first, capped: a genuinely offloaded model has thousands of
+ # these and the report is read by a human.
+ "off_gpu_parameters": sorted(off_gpu, key = lambda p: -p["numel"])[:20],
+ "off_gpu_parameter_count": len(off_gpu),
"hf_device_map_devices": (
sorted({str(v) for v in device_map.values()}) if isinstance(device_map, dict) else None
),
@@ -208,6 +251,91 @@ def build_dataset(tokenizer, rows: list[dict]):
return Dataset.from_dict({"text": texts})
+def build_completion_dataset(tokenizer, rows: list[dict]):
+ """The same rows as a PROMPT/COMPLETION pair, so the loss covers only the
+ answer.
+
+ Two columns rather than a collator: TRL treats a dataset with `prompt` and
+ `completion` columns as prompt-completion and masks the prompt itself
+ (`completion_only_loss` defaults to True for that shape). The older
+ `DataCollatorForCompletionOnlyLM` route needs a response template string
+ that has to match the chat template exactly, and a template change turns it
+ silently into "mask nothing" -- which trains on everything and passes every
+ assertion about losses.
+
+ The prompt ends with the generation prompt, so the boundary is exactly where
+ the model would start generating. That is what makes the mask meaningful
+ rather than approximately right.
+ """
+ from datasets import Dataset
+
+ prompts, completions = [], []
+ for row in rows:
+ prompts.append(
+ tokenizer.apply_chat_template(
+ [{"role": "user", "content": row["question"]}],
+ tokenize = False,
+ add_generation_prompt = True,
+ )
+ )
+ completions.append(row["answer"])
+ return Dataset.from_dict({"prompt": prompts, "completion": completions})
+
+
+def masking_evidence(trainer) -> dict:
+ """Whether the prompt tokens are ACTUALLY masked out of the loss.
+
+ This is the whole point of the feature and the one thing a loss curve
+ cannot show: a run that masks nothing trains on prompt and answer alike,
+ converges perfectly well, and reports numbers indistinguishable from a
+ correct one. So the labels are read off a real collated batch.
+
+ Never raises: a diagnostic that kills the leg is worse than a missing one.
+ """
+ record: dict = {}
+ try:
+ batch = next(iter(trainer.get_train_dataloader()))
+ labels = batch["labels"]
+ total = int(labels.numel())
+ masked = int((labels == -100).sum())
+ record["label_tokens"] = total
+ record["masked_tokens"] = masked
+ record["masked_fraction"] = round(masked / total, 4) if total else None
+ record["columns"] = sorted(batch.keys())
+ except BaseException as exc: # noqa: BLE001
+ record["error"] = f"{type(exc).__name__}: {exc}"[:400]
+ return record
+
+
+def masking_failures(record: dict | None, *, expected: bool) -> list[str]:
+ """The rule, as a pure function, so it is checkable without a GPU."""
+ if not expected:
+ return []
+ if not record:
+ return ["completions-only training was requested but no masking evidence was collected"]
+ if record.get("error"):
+ return [f"could not read the collated labels: {record['error']}"]
+
+ total = record.get("label_tokens") or 0
+ masked = record.get("masked_tokens")
+ if not total:
+ return ["the collated batch carried no labels at all"]
+ if not masked:
+ # The failure this function exists for. Every loss-based assertion
+ # passes in this state.
+ return [
+ "completions-only training was requested and NOTHING was masked "
+ f"({masked} of {total} label tokens are -100), so the prompt is in "
+ f"the loss and this leg is not testing what it says it is"
+ ]
+ if masked >= total:
+ return [
+ f"every one of the {total} label tokens is masked, so there is no "
+ f"completion left to learn from"
+ ]
+ return []
+
+
def train_and_infer(args) -> dict:
import torch
from unsloth import FastLanguageModel
@@ -270,13 +398,22 @@ def train_and_infer(args) -> dict:
for line in Path(args.dataset).read_text(encoding = "utf-8").splitlines()
if line.strip()
]
- dataset = build_dataset(tokenizer, rows)
+ # Prompt/completion when the loss should cover only the answer, one `text`
+ # column otherwise. `dataset_text_field` must go with the second shape and
+ # NOT the first: naming a text field TRL cannot find is how a
+ # prompt-completion dataset silently falls back to training on everything.
+ if args.train_on_completions:
+ dataset = build_completion_dataset(tokenizer, rows)
+ else:
+ dataset = build_dataset(tokenizer, rows)
+ result["train_on_completions"] = bool(args.train_on_completions)
+ result["dataset_columns"] = sorted(dataset.column_names)
from trl import SFTConfig, SFTTrainer
config = SFTConfig(
output_dir = str(Path(args.outdir) / "trainer"),
- dataset_text_field = "text",
+ **({} if args.train_on_completions else {"dataset_text_field": "text"}),
per_device_train_batch_size = 1,
gradient_accumulation_steps = 1,
max_length = args.max_seq_length,
@@ -309,6 +446,13 @@ def train_and_infer(args) -> dict:
args = config,
)
+ # Read BEFORE training, off a real collated batch. After training the
+ # dataloader has been consumed and a re-created one is not necessarily the
+ # object the trainer used.
+ if args.train_on_completions:
+ result["masking"] = masking_evidence(trainer)
+ _log(f"masking {json.dumps(result['masking'])}")
+
# The precision after Unsloth's patches have had their say. On a T4 this
# should be float32 and NOT fp16; fp16 here means FORCE_FLOAT32 stopped
# firing and the infinities it prevents are back.
@@ -392,6 +536,87 @@ def train_and_infer(args) -> dict:
result["generated"] = generated
result["canary_found"] = CANARY in generated
result["memory_peak"] = memory()
+
+ # GGUF, LAST, because it merges a 20B checkpoint and the merge is the
+ # heaviest thing in the leg. Anything after it would be measuring a session
+ # that has just written ~28GB.
+ #
+ # ASK for q8_0, ACCEPT only mxfp4. That pairing looks backwards and is the
+ # only one that works, which cost a probe to learn.
+ #
+ # gpt-oss overrides the request, out loud:
+ #
+ # GPT-OSS does not support GGUF quantization (requested: q8_0).
+ # Overriding to MXFP4 format.
+ # GPT-OSS model - skipping additional quantizations
+ #
+ # The obvious response is to ask for mxfp4 directly. unsloth REJECTS it as
+ # an input -- measured on kernel unsloth-probe-gptoss-r3-832c85:
+ #
+ # Unsloth: Quant method = [mxfp4] not supported. Choose from below:
+ # [not_quantized] [fast_quantized] [quantized] [f32] [bf16] ...
+ #
+ # So the documented override is the ONLY route to an MXFP4 file, and the
+ # leg has to travel it. Accepting only mxfp4 is what keeps that honest: a
+ # run that somehow produced a real q8_0 would fail, which is right, because
+ # gpt-oss q8_0 is documented impossible.
+ if getattr(args, "export_gguf", False):
+ from gguf_export import export_gguf, llama_cpp_facts, run_gguf
+
+ install_log = ""
+ llama_dir = None
+ try:
+ import contextlib
+ import io
+
+ # After `import unsloth`, which has happened by now: unsloth_zoo's
+ # llama_cpp raises "Please install Unsloth via pip install unsloth!"
+ # if it is reached first. A probe already lost a session to that.
+ from unsloth_zoo.llama_cpp import install_llama_cpp
+
+ buffer = io.StringIO()
+ with contextlib.redirect_stdout(buffer):
+ returned = install_llama_cpp()
+ install_log = buffer.getvalue()
+ facts = llama_cpp_facts(install_log, returned)
+ llama_dir = facts.get("dir")
+ except BaseException as exc: # noqa: BLE001
+ facts = {"error": f"{type(exc).__name__}: {exc}"[:2000]}
+ _log(f"llama.cpp: {json.dumps(facts)}")
+
+ # NOT under args.outdir, and this is measured. `/kaggle/working` is
+ # 21.0GB total; the gpt-oss export consumes 27.6GB of transient disk
+ # (3 mxfp4 safetensors shards at 13.76GB plus the GGUF). Exporting
+ # there fails in 2.8s with
+ # Unsloth: Failed saving locally - no disk space left
+ # which is a disk fact wearing the costume of an export bug.
+ #
+ # `/tmp` on a Kaggle session is the overlay: 8656.9GB total, 1102.5GB
+ # free, measured on kernel unsloth-probe-disk. tempfile honours TMPDIR
+ # and lands there. The artifact keeps the RECORD -- path, size, seconds
+ # -- and not the 27.6GB, which nobody wants collected anyway.
+ gguf_dir = tempfile.mkdtemp(prefix = "gptoss_gguf_")
+ record = export_gguf(
+ model,
+ tokenizer,
+ gguf_dir,
+ quantization = args.gguf_quantization,
+ )
+ record["disk_free_gb_before"] = round(shutil.disk_usage(gguf_dir).free / 1e9, 1)
+ record["llama_cpp"] = facts
+ result["gguf_export"] = record
+ _log(f"gguf export: {json.dumps({k: v for k, v in record.items() if k != 'llama_cpp'})}")
+
+ # Deliberately NOT run through llama.cpp here, and the reason is
+ # measured: the prebuilt bundle is the -cpu build, and a 20B MXFP4 file
+ # on 4 vCPUs is not a few seconds of work. The Default leg already
+ # covers "the exported file runs" on a 610MB Q8_0, where the claim is
+ # affordable. Asserting the export and not the run is a smaller claim,
+ # and it is the one this leg can honestly make.
+ ggufs = record.get("ggufs") or []
+ result["gguf_ran"] = False
+ if ggufs:
+ _log(f"largest gguf: {ggufs[0]['path']} ({ggufs[0]['mb']} MB), not executed here")
_log(f"generated {generated!r}")
return result
@@ -427,13 +652,41 @@ def _placement_failures(placement: dict | None) -> list[str]:
device: n for device, n in counts.items() if not str(device).startswith("cuda")
}
if elsewhere:
- failures.append(
- f"parameters are off the GPU: {elsewhere} (all devices: {counts}). "
- f"This leg's result is that the 20B checkpoint fits and trains "
- f"wholly on one T4; a run that offloads to CPU, disk or meta is "
- f"not a slower version of that, it is a different run, and "
- f"accelerate's offload does not support training at all"
+ # The ONE tensor allowed off the card, and only on its own terms.
+ # `offload_embedding` puts the input embedding in RAM and hooks the
+ # lookup so ids go down and vectors come back up; measured saving
+ # 1.08GB on gpt-oss-20b. Accepting "cpu" wholesale would excuse a
+ # real spill, so this names the exact parameter AND requires the
+ # hook flag: an embedding that reached the CPU without them is a
+ # bug, and it looks identical in a device count.
+ embed = placement.get("input_embedding") or {}
+ deliberate = (
+ embed.get("offload_hooks_installed") is True
+ and str(embed.get("device", "")).startswith("cpu")
+ and embed.get("weight_name")
)
+ unexplained = [
+ p
+ for p in (placement.get("off_gpu_parameters") or [])
+ if not (deliberate and p.get("name") == embed.get("weight_name"))
+ ]
+ if unexplained or not (placement.get("off_gpu_parameters") or []):
+ named = (
+ ", ".join(
+ f"{p.get('name')} ({p.get('numel')} on {p.get('device')})"
+ for p in unexplained
+ )
+ or "the walk recorded no names"
+ )
+ failures.append(
+ f"parameters are off the GPU: {elsewhere} [{named}] (all devices: "
+ f"{counts}), and unsloth's deliberate embedding offload does not "
+ f"account for them (input_embedding {embed}). This leg's result is "
+ f"that the 20B checkpoint fits and trains wholly on one T4; a run "
+ f"that offloads to CPU, disk or meta is not a slower version of "
+ f"that, it is a different run, and accelerate's offload does not "
+ f"support training at all"
+ )
# The accelerate-side answer, which is absent on a healthy run and says
# `cpu`/`disk` when dispatch offloaded. Three-way: `placement()` always
# writes a bool, so anything else is a record this file cannot read.
@@ -455,6 +708,16 @@ def failures_for(result: dict, args) -> list[str]:
that costs a Kaggle session has to be checkable without one.
"""
failures: list[str] = []
+ if getattr(args, "export_gguf", False):
+ from gguf_export import export_failures
+
+ # MXFP4 only. gpt-oss refuses every other quantization by design, so a
+ # wider accept list would let a silently-overridden export pass as
+ # though the request had been honoured.
+ failures += export_failures(result.get("gguf_export"), accept_quantizations = ("mxfp4",))
+ failures += masking_failures(
+ result.get("masking"), expected = bool(getattr(args, "train_on_completions", False))
+ )
metrics = result.get("metrics") or []
if len(metrics) != args.max_steps:
failures.append(f"expected {args.max_steps} logged steps, got " f"{len(metrics)}")
@@ -619,6 +882,20 @@ def main() -> int:
ap.add_argument("--max-seq-length", type = int, default = 1024)
ap.add_argument("--lora-r", type = int, default = 8)
ap.add_argument("--max-new-tokens", type = int, default = 32)
+ ap.add_argument(
+ # On by default: the leg is specified to train on completions, and a
+ # flag that has to be remembered at every call site is one that will be
+ # forgotten at one of them.
+ "--train-on-completions",
+ dest = "train_on_completions",
+ action = "store_true",
+ default = True,
+ )
+ ap.add_argument("--no-train-on-completions", dest = "train_on_completions", action = "store_false")
+ ap.add_argument("--export-gguf", dest = "export_gguf", action = "store_true", default = False)
+ # q8_0 as the REQUEST. `mxfp4` is not an accepted input value; the override
+ # is the only way to reach that format. See the export block above.
+ ap.add_argument("--gguf-quantization", default = "q8_0")
ap.add_argument("--require-compile", dest = "require_compile", action = "store_true", default = True)
ap.add_argument("--no-require-compile", dest = "require_compile", action = "store_false")
ap.add_argument(
diff --git a/tests/kaggle/t4_smoke/run_grpo_t4.py b/tests/kaggle/t4_smoke/run_grpo_t4.py
index c4242ace2b9..c0b9bef3c28 100644
--- a/tests/kaggle/t4_smoke/run_grpo_t4.py
+++ b/tests/kaggle/t4_smoke/run_grpo_t4.py
@@ -114,16 +114,38 @@ def _texts(completions) -> list[str]:
return out
+# Chosen so the reward can never saturate. len/(len+SCALE) is strictly
+# increasing on [0, inf), maps into (0, 1), and equals 0.5 at SCALE characters,
+# so it stays informative at every length instead of only below a cap.
+_LENGTH_SCALE = 200.0
+
+
def reward_length(completions, **kwargs) -> list[float]:
- """Longer completions score higher, saturating at 200 characters.
+ """Longer completions score higher, without ever saturating.
Deterministic given the text, yet SENSITIVE to a group's diversity: a
- constant reward would zero `reward_std` on a healthy run and destroy the
- only instrument this leg has.
+ constant reward zeroes `reward_std` on a healthy run and destroys the only
+ instrument this leg has.
+
+ THAT IS NOT HYPOTHETICAL - this function used to be
+
+ min(len(t), 200) / 200.0
+
+ and the docstring above already claimed it was diversity-sensitive. It was
+ not. Kernels unsloth-probe-grpo-rep2-b03be8 and -rep3-bc3828 recorded
+ completions of 2534 to 3396 characters, an order of magnitude past the cap,
+ so every completion scored exactly 1.0, every group tied, and the leg failed
+ with `reward_std was zero on every step` - a red that looks like a broken
+ generation path and is really a reward that stopped measuring anything.
+ Two of three runs died this way, which made it the leg's dominant failure.
+
+ The cap was the whole problem: it was set BELOW the lengths the model
+ actually produces, so the one region it discriminated in was the one region
+ the run never visited.
"""
texts = _texts(completions)
SEEN_COMPLETIONS.append(texts)
- return [min(len(t), 200) / 200.0 for t in texts]
+ return [len(t) / (len(t) + _LENGTH_SCALE) for t in texts]
def reward_digit(completions, **kwargs) -> list[float]:
diff --git a/tests/kaggle/t4_smoke/run_t4_smoke.py b/tests/kaggle/t4_smoke/run_t4_smoke.py
index 6908c798673..d8ffd77b8c9 100644
--- a/tests/kaggle/t4_smoke/run_t4_smoke.py
+++ b/tests/kaggle/t4_smoke/run_t4_smoke.py
@@ -67,6 +67,7 @@ import json
import os
import platform
import subprocess
+import tempfile
import sys
import time
from pathlib import Path
@@ -84,6 +85,8 @@ from determinism import ( # noqa: E402
set_all_seeds_fast,
set_deterministic_algorithms,
)
+from kernel_provenance import attention_choice, probe_kernels, vision_kernel_failures # noqa: E402
+from naive_trl_compare import comparison_failures # noqa: E402
from training_evidence import LORA_B_MARKER, LORA_MARKER # noqa: E402
from versions import ( # noqa: E402
GOAL_PACKAGES,
@@ -266,11 +269,40 @@ def train_once(args, run_index: int) -> dict:
# float16 unconditionally: T4 is sm_75 and has no bf16. Pinned rather than
# left to the loader so the local reproduction and the Kaggle run take the
# same numeric path wherever they can share one.
+ load_kwargs: dict = {}
+ if getattr(args, "single_device", False):
+ # Both cards stay VISIBLE -- that is the point of the multi_gpu leg, and
+ # what makes unsloth's DEVICE_COUNT > 1 bindings live -- but the weights
+ # go on one, because a SHARDED model does not train.
+ #
+ # Measured on unsloth-probe-multigpu-r1-18beab: with two T4s visible
+ # accelerate split Qwen3-0.6B 232849408 params on cuda:0 and 155582464
+ # on cuda:1, unsloth printed `Num GPUs used = 2`, and step 0 died in
+ # unsloth/models/llama.py:972 --
+ #
+ # inputs_embeds = self.embed_tokens(input_ids)
+ # RuntimeError: Expected all tensors to be on the same device, but got
+ # index is on cuda:0, different from other tensors on cuda:1
+ #
+ # WHETHER THAT IS A BUG IS NOT SETTLED, and an earlier version of this
+ # comment claimed it was filed. It is not filed, and the repo's own
+ # history is split on it: #2467 was closed with "unsloth runs on only 1
+ # gpu.. It wont run in multi gpu systems. force it to use only 1 gpu",
+ # while #2882 -- the same message, a different call site -- was closed
+ # as fixed. So this may be documented behaviour rather than a defect.
+ #
+ # What is not in doubt is that it is not THIS repo's to work around in a
+ # per-PR check. Training a sharded model here would put a red in front
+ # of every PR for a fault no reader can act on, which is how a check
+ # gets switched off before the day it is right. The binding coverage --
+ # the whole reason this leg exists -- does not depend on it.
+ load_kwargs["device_map"] = {"": 0}
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = args.model,
max_seq_length = args.max_seq_length,
load_in_4bit = True,
dtype = torch.float16,
+ **load_kwargs,
)
load_seconds = time.time() - t0
# Which repository, and which commit of it. `load_in_4bit=True` redirects
@@ -284,6 +316,26 @@ def train_once(args, run_index: int) -> dict:
resolved_revision = getattr(_config, "_commit_hash", None)
_log(f"loaded {resolved_checkpoint} @ {resolved_revision}")
+ # AFTER the load and not before, which is measured rather than tidy: on
+ # kernel unsloth-probe-vision-recon-c76ea3 `fla` was NOT importable before
+ # from_pretrained and WAS after, because unsloth reaches for it lazily when
+ # it sees the model. Reading provenance up front reports it absent.
+ if getattr(args, "kernel_provenance", False):
+ result_kernels = probe_kernels()
+ result_attention = attention_choice(model)
+ _log(f"kernels: {json.dumps(result_kernels)}")
+ _log(f"attention: {json.dumps(result_attention)}")
+ else:
+ result_kernels = None
+ result_attention = None
+
+ # AFTER the load for the same reason as the block above: the rotary caches
+ # are built when the model is, so reading them before from_pretrained finds
+ # nothing and reports it as an absence.
+ result_multi_gpu = multi_gpu_facts(model) if getattr(args, "require_multi_gpu", False) else None
+ if result_multi_gpu is not None:
+ _log(f"multi-gpu: {json.dumps(result_multi_gpu)}")
+
# Bound to a name so the saved config is checked against the adapter that
# was actually requested, rather than against a list repeated further down.
target_modules = [
@@ -401,7 +453,16 @@ def train_once(args, run_index: int) -> dict:
# function of the weights alone.
FastLanguageModel.for_inference(model)
prompt = PROMPT_TEMPLATE.format(question = rows[0]["question"])
- inputs = tokenizer([prompt], return_tensors = "pt").to(model.device)
+ # `text = ` and not positional, and this cost a leg. A vision model's
+ # tokenizer IS a processor, whose signature is
+ # `__call__(self, images=None, text=None, videos=None, ...)` -- so a
+ # positional list is taken as IMAGES and transformers tries to fetch the
+ # prompt strings as image URLs. Measured on kernel
+ # unsloth-probe-latestcompile-r2-62b54d with gemma-4-E2B-it:
+ # transformers/image_processing_backends.py, in fetch_images
+ # `text` is the first parameter of a plain tokenizer too, so the keyword is
+ # correct for both and is not a special case for vision.
+ inputs = tokenizer(text = [prompt], return_tensors = "pt").to(model.device)
t0 = time.time()
with torch.inference_mode():
out = model.generate(
@@ -417,16 +478,123 @@ def train_once(args, run_index: int) -> dict:
infer_seconds = time.time() - t0
generated = tokenizer.decode(out[0][inputs["input_ids"].shape[1] :], skip_special_tokens = True)
+ # Batched generation, on the SAME trained model, immediately after the
+ # single-prompt generation above. Prompts of deliberately different lengths,
+ # because a batch of equal-length prompts pads nothing and would report a
+ # green left-padding check that never padded.
+ batch_prompts = [
+ PROMPT_TEMPLATE.format(question = row["question"]) for row in rows[: max(BATCH_SIZES)]
+ ]
+ while len(batch_prompts) < max(BATCH_SIZES):
+ # The canary dataset is small. Pad the LIST (not the tensors) by
+ # reusing questions with a varying prefix, which keeps the token
+ # lengths spread rather than repeating one length.
+ idx = len(batch_prompts)
+ batch_prompts.append(
+ PROMPT_TEMPLATE.format(
+ question = " ".join(["please"] * (idx % 5 + 1))
+ + " "
+ + rows[idx % len(rows)]["question"]
+ )
+ )
+ batched = batched_generation(
+ model,
+ tokenizer,
+ batch_prompts,
+ max_new_tokens = args.max_new_tokens,
+ )
+ _log(f"batched generation: {json.dumps({k: v for k, v in batched.items() if k != 'batched'})}")
+
+ # GGUF export, opt-in. Placed AFTER generation deliberately: the export
+ # merges the adapter into the base weights, and doing that before the
+ # canary and batched-generation checks would have them measure a different
+ # model from the one training produced.
+ gguf_export_record = None
+ gguf_run_record = None
+ # ONCE per leg, on the first cycle, not once per cycle. Measured: the
+ # conversion is 310.8s and 312.3s on the two Latest_compile cycles and
+ # 99.3s and 117.3s on the two vision ones, so the repeat is 47% of the
+ # longest leg in the suite. The second export is the same base weights and
+ # an adapter trained by the same script with the same seed, so it re-runs
+ # llama.cpp rather than asking a new question; reproducibility is already
+ # asserted on the step tables and on the generated text.
+ if getattr(args, "export_gguf", False) and run_index > 0:
+ gguf_export_record = {
+ "skipped": "exported on cycle 0; the conversion is the same weights twice"
+ }
+ elif getattr(args, "export_gguf", False):
+ from gguf_export import export_gguf, llama_cpp_facts, run_gguf
+
+ # unsloth must be imported before unsloth_zoo.llama_cpp, which raises
+ # "Please install Unsloth via pip install unsloth!" otherwise. It is,
+ # by the time train_once runs, but the import stays local so a payload
+ # that never exports does not pay for it.
+ install_log = ""
+ llama_dir = None
+ try:
+ import contextlib
+ import io
+
+ from unsloth_zoo.llama_cpp import install_llama_cpp
+
+ buffer = io.StringIO()
+ with contextlib.redirect_stdout(buffer):
+ returned = install_llama_cpp()
+ install_log = buffer.getvalue()
+ facts = llama_cpp_facts(install_log, returned)
+ llama_dir = facts.get("dir")
+ except BaseException as exc: # noqa: BLE001
+ facts = {"error": f"{type(exc).__name__}: {exc}"[:2000]}
+ _log(f"llama.cpp: {json.dumps(facts)}")
+
+ # NOT under args.outdir. That is `/kaggle/working`, which is 21.0 GB
+ # and is also what `kernels output` ships back, so a merge that fits
+ # there is downloaded and a merge that does not kills the leg. Measured
+ # on unsloth-probe-lcleg-final-a90fbb:
+ #
+ # RuntimeError: Unsloth: Not enough disk space to convert to GGUF.
+ # The export needs about 16.6GB on the filesystem holding
+ # `/kaggle/working/t4_out_Latest_compile/cycle0/gguf_run0`
+ #
+ # /tmp is the same overlay as `/`, 8656.9 GB with ~1100 GB free, which
+ # this file has recorded since the first disk probe. tempfile honours
+ # TMPDIR and lands there. The RECORD travels back -- path, size,
+ # seconds -- and the multi-gigabyte artifact does not, which nobody
+ # wanted collected anyway.
+ #
+ # gpt-oss and the vision run were both moved for this exact reason and
+ # this path was missed, which is why a small model kept passing.
+ gguf_export_record = export_gguf(
+ model,
+ tokenizer,
+ tempfile.mkdtemp(prefix = f"gguf_run{run_index}_"),
+ quantization = args.gguf_quantization,
+ )
+ gguf_export_record["llama_cpp"] = facts
+ _log(
+ f"gguf export: {json.dumps({k: v for k, v in gguf_export_record.items() if k != 'llama_cpp'})}"
+ )
+
+ ggufs = gguf_export_record.get("ggufs") or []
+ if ggufs and llama_dir:
+ gguf_run_record = run_gguf(ggufs[0]["path"], llama_dir)
+
peak_gb = torch.cuda.max_memory_reserved() / 1024**3 if torch.cuda.is_available() else 0.0
result = {
"run_index": run_index,
+ "kernels": result_kernels,
+ "attention": result_attention,
+ "multi_gpu": result_multi_gpu,
"metrics": stats.logs,
"generated": generated,
# Both, because they answer different questions when this goes red:
# `canary_found` says training reached the weights at all, `canary_exact`
# is the assertion, and the gap between them is the signature of a
# stopping/EOS regression rather than a training one.
+ "batched_generation": batched,
+ "gguf_export": gguf_export_record,
+ "gguf_run": gguf_run_record,
"canary_found": CANARY in generated,
"canary_exact": generated.strip() == CANARY,
"prompt": prompt,
@@ -511,6 +679,27 @@ def _reconstruct_adapter_config(adapter_dir, expected: dict | None) -> dict:
unchecked.append(key)
continue
got = getattr(config, key)
+ if key == "target_modules" and isinstance(got, str):
+ # PEFT allows target_modules to be a REGEX as well as a list, and
+ # unsloth writes one for vision models so the adapter targets the
+ # language tower and not the vision encoder. Measured on
+ # unsloth-probe-vision-leg-r2-793ec0 with Qwen3.5-2B and again on
+ # gemma-4-E2B-it, where the saved value begins
+ # (?:.*?(?:language|text).*?(?:self_attn|attention|...
+ #
+ # Comparing that string against the list that was REQUESTED reports
+ # a difference on every vision model, which is correct behaviour
+ # being called a defect. The claim worth keeping is narrower and
+ # still falsifiable: every module name asked for must appear in the
+ # pattern, so a silently dropped projection is still caught.
+ missing = [name for name in (wanted or []) if name not in got]
+ same = not missing
+ if not same:
+ differences.append(
+ f"{key}: trained with {wanted!r}, and the saved regex does "
+ f"not mention {missing!r}: {got!r}"
+ )
+ continue
if isinstance(wanted, (list, tuple, set)) or isinstance(got, (list, tuple, set)):
same = sorted(got or []) == sorted(wanted or [])
else:
@@ -522,6 +711,307 @@ def _reconstruct_adapter_config(adapter_dir, expected: dict | None) -> dict:
return out
+# Batch sizes to cross-check against one-at-a-time generation. 1 is the
+# baseline and is generated separately; the rest must reproduce it exactly.
+BATCH_SIZES = (2, 4, 8)
+
+
+def batched_generation(model, tokenizer, prompts, *, max_new_tokens) -> dict:
+ """Greedy generation one-at-a-time, then batched, and whether they agree.
+
+ WHAT THIS IS FOR. Batched generation with left padding has broken here
+ before, repeatedly and in ways that pass every other check in this file:
+
+ * #3699 batched generation with left-padding and caching produced incorrect
+ output,
+ * #1066 batch inference produced gibberish,
+ * #1456 batch inference was inconsistent for a self-trained model,
+ * #2138 a release silently FORCED the tokenizer padding side to right during
+ inference, which is why the side is recorded as OBSERVED after generating
+ rather than as the value this function set.
+
+ Greedy decoding makes the comparison meaningful: the output is then a
+ function of the weights and the attention mask alone, so any difference
+ between batch sizes is padding or cache handling rather than sampling.
+
+ THE VACUITY TRAP, and it is the whole reason this returns the token lengths:
+ padding only happens when the prompts in a batch have DIFFERENT lengths.
+ A batch of equal-length prompts pads nothing, agrees trivially, and reports
+ a green left-padding check that never once left-padded. The caller asserts
+ the spread; this function measures it.
+ """
+ # Imported here, not at module scope, matching every other torch user in
+ # this file: the module is loaded before unsloth is installed in some
+ # paths, and a top-level torch import would move the failure to import
+ # time. Kernel unsloth-probe-defaultleg-723c28 trained all 10 steps and
+ # then died on `NameError: name 'torch' is not defined` right here.
+ import torch
+
+ tokenizer.padding_side = "left"
+ if tokenizer.pad_token is None:
+ tokenizer.pad_token = tokenizer.eos_token
+
+ def _gen(batch: list) -> list:
+ # Keyword for the same reason as the single-prompt path above: a
+ # processor reads a positional list as images.
+ enc = tokenizer(text = batch, return_tensors = "pt", padding = True).to(model.device)
+ with torch.inference_mode():
+ out = model.generate(
+ **enc,
+ max_new_tokens = max_new_tokens,
+ do_sample = False,
+ temperature = None,
+ top_p = None,
+ top_k = None,
+ use_cache = True,
+ # `x or y` is wrong here: pad_token_id 0 is a perfectly ordinary
+ # id (Qwen and Llama both use low ids) and is falsy, so `or`
+ # silently substitutes the EOS id for it and pads the batch with
+ # end-of-sequence tokens. Test for None.
+ pad_token_id = (
+ tokenizer.pad_token_id
+ if tokenizer.pad_token_id is not None
+ else tokenizer.eos_token_id
+ ),
+ )
+ # Slice by the PADDED width, not by each prompt's own length: with left
+ # padding every row starts at the same column, and using the unpadded
+ # length would re-read the tail of the prompt as if it were output.
+ width = enc["input_ids"].shape[1]
+ return [tokenizer.decode(row[width:], skip_special_tokens = True) for row in out]
+
+ # `[0]`, and the missing index made this whole check vacuous. A processor
+ # returns input_ids with a BATCH dimension, so `len(...)` on it is the
+ # number of sequences -- 1 -- for every prompt. Measured on kernel
+ # unsloth-probe-latestcompile-r3-cb1125, where gemma-4 reported
+ # [1, 1, 1, 1, 1, 1, 1, 1] and the vacuity guard below caught it:
+ # "every batched prompt tokenised to the same length ... so nothing was
+ # ever padded and the left-padding check proved nothing"
+ # A plain tokenizer given one string returns a flat list, which is why this
+ # read correctly on every text model and only broke on the first vision one.
+ lengths = [len(tokenizer(text = [p])["input_ids"][0]) for p in prompts]
+ singles = [_gen([p])[0] for p in prompts]
+ result = {
+ "prompt_token_lengths": lengths,
+ "distinct_lengths": len(set(lengths)),
+ "padding_side_observed": tokenizer.padding_side,
+ "singles": singles,
+ "batched": {},
+ "agrees": {},
+ "empty_outputs": [i for i, text in enumerate(singles) if not text.strip()],
+ }
+ for size in BATCH_SIZES:
+ outs: list = []
+ for start in range(0, len(prompts), size):
+ outs.extend(_gen(prompts[start : start + size]))
+ result["batched"][str(size)] = outs
+ result["agrees"][str(size)] = outs == singles
+ # Read AGAIN, after all the generating. #2138 was a silent override applied
+ # inside the inference path, so the value set at the top of this function is
+ # not evidence of the value that was used.
+ result["padding_side_after"] = tokenizer.padding_side
+ return result
+
+
+# Models where batched greedy generation is CONFIRMED broken upstream, filed,
+# and not this repo's to fix. See unsloth #9708: on both of these, batch sizes
+# 2/4/8 fail to reproduce one-at-a-time output with real left padding and
+# demonstrably distinct prompt lengths, on both repeats.
+#
+# This is a STRICT expectation, not a mute. A model listed here that starts
+# AGREEING fails the leg, with a message saying to delete the entry -- so the
+# day #9708 is fixed, CI says so instead of quietly keeping a stale excuse.
+# Every other rule in this function stays live for these models: the padding
+# side, the distinct lengths and the empty-output check are what make the
+# disagreement a real finding rather than an unpadded batch.
+KNOWN_BATCHED_GENERATION_BREAKAGE = {
+ "unsloth/gemma-4-E2B-it": "unsloth#9708",
+ "unsloth/Qwen3.5-2B": "unsloth#9708",
+}
+
+
+def batched_generation_failures(batch: dict | None, model: str | None = None) -> list[str]:
+ """Turn a `batched_generation` record into failures, vacuity included."""
+ if not batch:
+ return ["batched generation was never run"]
+ out = []
+ known = KNOWN_BATCHED_GENERATION_BREAKAGE.get(model or "")
+ if batch.get("distinct_lengths", 0) < 2:
+ out.append(
+ "every batched prompt tokenised to the same length "
+ f"({batch.get('prompt_token_lengths')}), so nothing was ever padded "
+ "and the left-padding check proved nothing"
+ )
+ if len(batch.get("singles") or []) < max(BATCH_SIZES):
+ out.append(
+ f"only {len(batch.get('singles') or [])} prompts for a batch size of "
+ f"{max(BATCH_SIZES)}, so the largest batch was never actually formed"
+ )
+ for side_key in ("padding_side_observed", "padding_side_after"):
+ if batch.get(side_key) != "left":
+ out.append(
+ f"{side_key} is {batch.get(side_key)!r}, not 'left'; a right-padded "
+ f"decoder-only batch attends to pad tokens before the prompt (#2138)"
+ )
+ if batch.get("empty_outputs"):
+ out.append(f"prompts {batch['empty_outputs']} generated nothing at all")
+ agrees = batch.get("agrees") or {}
+ for size, agreed in agrees.items():
+ if not agreed and not known:
+ out.append(
+ f"batch size {size} did not reproduce one-at-a-time greedy output "
+ f"(#3699/#1456): {batch.get('batched', {}).get(size)!r} != "
+ f"{batch.get('singles')!r}"
+ )
+ if known and agrees and all(agrees.values()):
+ # The strict half. An expectation that only ever excuses is a mute, and
+ # a mute outlives the bug it was written for.
+ out.append(
+ f"{model} is listed in KNOWN_BATCHED_GENERATION_BREAKAGE for "
+ f"{known}, and every batch size AGREED. If the upstream fix has "
+ f"landed, delete the entry; the leg must not carry an excuse for a "
+ f"bug that is gone"
+ )
+ return out
+
+
+# Set by multi_gpu_facts as soon as it has an answer, so a crash LATER in the
+# cycle still reports what was measured. A reading that exists only inside a
+# frame that is being unwound is a reading nobody has.
+_LAST_MULTI_GPU_FACTS: dict | None = None
+
+
+def multi_gpu_facts(model) -> dict:
+ """What unsloth BOUND, given how many cards this process can see.
+
+ The point of the multi_gpu leg is a branch no pinned payload can reach.
+ `unsloth/kernels/utils.py:170`:
+
+ if DEVICE_COUNT > 1:
+ torch_gpu_device = torch.cuda.device # a real device switch
+ else:
+ def torch_gpu_device(device): return nullcontext()
+
+ `build_kernel.py` pins every ordinary payload with CUDA_VISIBLE_DEVICES, so
+ every unsloth kernel this CI has run took the nullcontext branch, and so did
+ the DEVICE_COUNT-sized CUDA_STREAMS / WEIGHT_BUFFERS / ABSMAX_BUFFERS arrays
+ and the per-device rotary caches in unsloth/models/llama.py.
+
+ Read off the IMPORTED MODULE, not recomputed from `device_count()`. The
+ binding is made once at import time, so asking torch how many cards there
+ are answers a different question -- and answers it the way the check wants,
+ which is the shape of a rule that cannot fail.
+ """
+ import torch
+
+ global _LAST_MULTI_GPU_FACTS
+ facts: dict = {"device_count": torch.cuda.device_count()}
+ _LAST_MULTI_GPU_FACTS = facts
+ try:
+ from unsloth.kernels import utils as _kernel_utils
+ except BaseException as exc: # noqa: BLE001
+ facts["error"] = f"{type(exc).__name__}: {exc}"
+ return facts
+
+ binding = getattr(_kernel_utils, "torch_gpu_device", None)
+ facts["module_device_count"] = getattr(_kernel_utils, "DEVICE_COUNT", None)
+ # `torch.cuda.device` is a CLASS; the single-card fallback is a module-level
+ # function closing over nullcontext. Identity against the real thing rather
+ # than a name check, because both are bound to the same name.
+ facts["torch_gpu_device_is_real_switch"] = binding is torch.cuda.device
+ facts["torch_gpu_device_repr"] = repr(binding)[:200]
+ for name in ("CUDA_STREAMS", "WEIGHT_BUFFERS", "ABSMAX_BUFFERS"):
+ value = getattr(_kernel_utils, name, None)
+ facts[name.lower() + "_len"] = None if value is None else len(value)
+
+ # Where the weights actually sit. With two cards visible accelerate may
+ # shard, and whether it does is a MEASUREMENT rather than something to
+ # assert blind: kernel unsloth-probe-vision-recon-c76ea3 saw a model split
+ # 897.7 MB / 1017.1 MB across two T4s, but that was a different model and a
+ # different loader path.
+ by_device: dict[str, int] = {}
+ try:
+ for _, param in model.named_parameters():
+ key = str(param.device)
+ by_device[key] = by_device.get(key, 0) + param.numel()
+ except BaseException as exc: # noqa: BLE001
+ facts["parameter_walk_error"] = f"{type(exc).__name__}: {exc}"
+ facts["parameters_by_device"] = by_device
+ facts["cuda_devices_holding_parameters"] = sorted(d for d in by_device if d.startswith("cuda"))
+
+ # The per-device rotary caches, sized by DEVICE_COUNT at construction. A
+ # list of length 1 on a two-card box means the model was built while
+ # unsloth believed there was one card.
+ for module in getattr(model, "modules", lambda: [])():
+ cached = getattr(module, "multi_gpu_cos_cached", None)
+ if cached is not None:
+ facts["rotary_cache_slots"] = len(cached)
+ break
+ return facts
+
+
+def multi_gpu_failures(facts: dict | None, *, expected_cards: int) -> list[str]:
+ """The rules, separated from the reading so they can be driven on CPU.
+
+ Deliberately NOT asserting that the parameters are spread across both
+ cards. Whether unsloth shards them or pins them to cuda:0 is exactly what
+ this leg is being run to find out, and a rule written before the answer is
+ a rule written to match whatever happens.
+ """
+ if not facts:
+ return [
+ "the multi-GPU facts are missing, so nothing about the "
+ "DEVICE_COUNT > 1 path was measured on this run"
+ ]
+ if facts.get("error"):
+ return [f"unsloth.kernels.utils could not be read: {facts['error']}"]
+
+ out: list[str] = []
+ seen = facts.get("device_count")
+ if seen != expected_cards:
+ out.append(
+ f"this leg exists to exercise the multi-card path and torch sees "
+ f"{seen} card(s), not {expected_cards} -- the driver pinned it, so "
+ f"every assertion below would measure the single-card branch under "
+ f"a multi-GPU name"
+ )
+ # Everything after this measures the wrong machine, so say so once.
+ return out
+
+ if facts.get("module_device_count") != expected_cards:
+ out.append(
+ f"unsloth.kernels.utils.DEVICE_COUNT is "
+ f"{facts.get('module_device_count')!r} while torch sees "
+ f"{expected_cards}; the module was imported before the cards were "
+ f"visible, so its bindings are the single-card ones"
+ )
+ if not facts.get("torch_gpu_device_is_real_switch"):
+ out.append(
+ f"torch_gpu_device is not torch.cuda.device but "
+ f"{facts.get('torch_gpu_device_repr')!r} -- the nullcontext "
+ f"fallback, which performs NO device switch, so this run covered "
+ f"the same path a pinned leg already covers"
+ )
+ for name in ("cuda_streams", "weight_buffers", "absmax_buffers"):
+ length = facts.get(name + "_len")
+ if length is None or length < expected_cards:
+ out.append(
+ f"unsloth.kernels.utils.{name.upper()} has {length!r} entries "
+ f"for {expected_cards} cards, so a dequant on the second card "
+ f"has no stream or buffer of its own"
+ )
+ slots = facts.get("rotary_cache_slots")
+ if slots is not None and slots < expected_cards:
+ out.append(
+ f"the per-device rotary cache has {slots} slot(s) for "
+ f"{expected_cards} cards, so the model was built while unsloth "
+ f"believed there was one"
+ )
+ if not facts.get("cuda_devices_holding_parameters"):
+ out.append("no parameter is on a CUDA device at all")
+ return out
+
+
def peft_adapter_keys(model) -> dict:
"""The names PEFT itself gives this adapter's tensors, off the live model.
@@ -1278,6 +1768,38 @@ def optimisation_failures(metrics: list[dict]) -> list[str]:
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--model", default = DEFAULT_MODEL)
+ # On by default: batched generation is the surface that has broken most
+ # often here (#3699, #1066, #1456, #2138) and a leg that quietly skips it
+ # is a leg that stops covering it.
+ ap.add_argument(
+ "--check-batched-generation",
+ dest = "check_batched_generation",
+ action = "store_true",
+ default = True,
+ )
+ ap.add_argument(
+ "--no-check-batched-generation",
+ dest = "check_batched_generation",
+ action = "store_false",
+ )
+ # OFF by default, unlike the batched-generation check above. That one is
+ # pure compute on a model already in memory; this one installs llama.cpp
+ # and merges the adapter, about 40s for a 0.6B on top of a ~10-46s install,
+ # so a leg opts in rather than every payload paying for it.
+ ap.add_argument(
+ "--export-gguf",
+ dest = "export_gguf",
+ action = "store_true",
+ default = False,
+ )
+ # What the exported filename is allowed to say. More than one because a
+ # model may legitimately override the request: gpt-oss answers q8_0 with
+ # "Overriding to MXFP4 format" by design, and failing on documented
+ # behaviour would be a failure invented rather than found.
+ ap.add_argument("--gguf-quantization", default = "q8_0")
+ ap.add_argument(
+ "--gguf-accept", default = "", help = "comma separated; defaults to the requested one"
+ )
ap.add_argument("--dataset", default = str(_HERE / "canary_dataset.jsonl"))
ap.add_argument("--outdir", required = True)
# 3 steps, and the whole reason --init-loss-scale exists.
@@ -1367,6 +1889,76 @@ def main() -> int:
)
ap.add_argument("--require-canary", dest = "require_canary", action = "store_true", default = True)
ap.add_argument("--no-require-canary", dest = "require_canary", action = "store_false")
+ ap.add_argument(
+ # The plain-TRL control arm, in its own process. Off by default: it
+ # doubles the leg's train time and only one leg is asking the question.
+ "--compare-naive-trl",
+ action = "store_true",
+ default = False,
+ help = "also train the same rows with plain TRL and report both traces",
+ )
+ ap.add_argument(
+ # The multi-card leg, and it is the ONLY thing that makes unsloth's
+ # DEVICE_COUNT > 1 bindings reachable. build_kernel.py pins every other
+ # payload with CUDA_VISIBLE_DEVICES, so `torch_gpu_device` is the
+ # nullcontext shim in every run this CI has ever produced.
+ "--require-multi-gpu",
+ dest = "require_multi_gpu",
+ action = "store_true",
+ default = False,
+ help = "assert unsloth bound its multi-card code path, and record where "
+ "the weights landed",
+ )
+ ap.add_argument(
+ # Keep every card VISIBLE and put the WEIGHTS on one. See the comment at
+ # the from_pretrained call: sharded training is broken upstream, and the
+ # DEVICE_COUNT > 1 bindings this leg exists to cover do not need it.
+ "--single-device",
+ dest = "single_device",
+ action = "store_true",
+ default = False,
+ help = "load with device_map={'': 0} while leaving both cards visible",
+ )
+ ap.add_argument(
+ # How many cards this leg was BUILT for, so the check compares against a
+ # declaration rather than against whatever the session happened to give
+ # it. Reading device_count() on both sides of the comparison is how a
+ # rule ends up unable to fail.
+ "--expected-cards",
+ dest = "expected_cards",
+ type = int,
+ default = 2,
+ )
+ ap.add_argument(
+ # Off by default: only the vendored-kernel leg is asking, and probing
+ # imports in every leg would add imports to legs that never wanted them.
+ "--kernel-provenance",
+ dest = "kernel_provenance",
+ action = "store_true",
+ default = False,
+ help = "record which fast kernels loaded and where each resolved from",
+ )
+ ap.add_argument(
+ # A SEPARATE PROCESS, like the plain-TRL control and for a related
+ # reason: the vision run loads a second model, and two 4bit models
+ # resident at once on a 14.56GB T4 is how a leg becomes an OOM blamed
+ # on the thing it was testing. It also runs AFTER the cycles, so a
+ # vision failure cannot be mistaken for a text-training one.
+ "--vision-run",
+ dest = "vision_run",
+ action = "store_true",
+ default = False,
+ help = "also drive run_vision_t4.py and fold its verdict into this report",
+ )
+ ap.add_argument(
+ # Only for models the control arm demonstrably cannot LOAD on the card.
+ # An OOM during training stays a failure either way.
+ "--control-oom-is-ok",
+ dest = "control_oom_is_ok",
+ action = "store_true",
+ default = False,
+ help = "a plain-TRL OOM before the first step is reported, not failed",
+ )
ap.add_argument("--label", default = "t4-smoke")
args = ap.parse_args()
@@ -1375,7 +1967,29 @@ def main() -> int:
# Child mode: exactly one cycle, report to disk, no assertions.
if args.cycle >= 0:
- run = train_once(args, args.cycle)
+ try:
+ run = train_once(args, args.cycle)
+ except BaseException as exc:
+ # A crash mid-cycle must not take the readings taken BEFORE it down
+ # with it. On kernel unsloth-probe-multigpu-r1-18beab the multi-card
+ # facts were gathered, logged in full, and then lost: the cycle died
+ # in trainer.train() and wrote no report at all, so the leg reported
+ # `multi_gpu: null` while the driver log held every number. Reading
+ # the report alone said the measurement had not been taken.
+ #
+ # The partial report carries the error, so it can never be mistaken
+ # for a completed cycle: `cycle_error` is what the parent's own
+ # "cycle N did not complete" failure is already keyed on.
+ partial = {
+ "run_index": args.cycle,
+ "cycle_error": f"{type(exc).__name__}: {exc}"[:2000],
+ "partial": True,
+ "multi_gpu": _LAST_MULTI_GPU_FACTS,
+ }
+ (outdir / "cycle_report.json").write_text(
+ json.dumps(partial, indent = 2), encoding = "utf-8"
+ )
+ raise
for entry in run["metrics"]:
_log(
f" step {entry['step']} loss={entry['loss']!r} "
@@ -1450,8 +2064,33 @@ def main() -> int:
("--gradient-checkpointing", args.gradient_checkpointing),
("--max-new-tokens", args.max_new_tokens),
("--label", args.label),
+ # The export settings travel too. Forgetting them is not a
+ # hypothetical: --export-gguf reached the PARENT on kernel
+ # unsloth-probe-default-gguf-637565, was parsed there, and never
+ # reached the child that actually runs train_once, so every cycle
+ # reported `gguf_export: null` and the leg failed with "GGUF export
+ # was never run" while the driver log plainly showed the flag on
+ # the command line.
+ ("--gguf-quantization", args.gguf_quantization),
+ ("--gguf-accept", args.gguf_accept),
+ # The CYCLE is what loads the model, so it is the only process that
+ # can read where unsloth bound its multi-card helpers. A parent that
+ # parsed this and kept it would report `multi_gpu: null` for every
+ # cycle and the leg would pass having measured nothing.
+ ("--expected-cards", args.expected_cards),
):
cmd += [flag, str(value)]
+ if args.export_gguf:
+ cmd.append("--export-gguf")
+ # Forwarded explicitly. A flag parsed in the parent and never passed on
+ # is not hypothetical here: --export-gguf did exactly that on kernel
+ # unsloth-probe-default-gguf-637565 and every cycle reported null.
+ if args.kernel_provenance:
+ cmd.append("--kernel-provenance")
+ if args.require_multi_gpu:
+ cmd.append("--require-multi-gpu")
+ if args.single_device:
+ cmd.append("--single-device")
if args.force_sdpa:
cmd.append("--force-sdpa")
if args.strict_deterministic:
@@ -1478,6 +2117,101 @@ def main() -> int:
return 1
runs.append(json.loads(report_file.read_text(encoding = "utf-8")))
+ # The plain-TRL control arm. AFTER the cycles, not before and not beside:
+ # it wants the same card, and two 4bit models resident at once on a 14.56GB
+ # T4 is how a comparison turns into an OOM blamed on the thing being
+ # compared. A fresh process because unsloth patches transformers, trl and
+ # peft at import; this parent has never imported it (only the cycle
+ # children do), but relying on that would make the control's validity a
+ # property of import order in a file nobody reads for that.
+ naive = None
+ if args.compare_naive_trl:
+ naive_dir = outdir / "naive_trl"
+ naive_dir.mkdir(parents = True, exist_ok = True)
+ _log("=== plain-TRL control arm (fresh process, unsloth not imported) ===")
+ cmd = [
+ sys.executable,
+ str(Path(__file__).resolve().parent / "naive_trl_compare.py"),
+ "--outdir",
+ str(naive_dir),
+ ]
+ # The repo UNSLOTH RESOLVED, not the name that was asked for. This is
+ # the difference between a comparison and an OOM: `load_in_4bit=True`
+ # sends unsloth through FLOAT_TO_INT_MAPPER to a pre-quantised
+ # `-unsloth-bnb-4bit` sibling, while the plain path quantises the
+ # ORIGINAL on the fly and therefore has to materialise the 16bit
+ # checkpoint first. On gemma-4-E2B-it that asked for 8.75GiB on top of
+ # 7.25GiB already resident and died (kernel
+ # unsloth-probe-latestcompile-r3-cb1125).
+ #
+ # Pointing both arms at the same weights is also the fairer comparison:
+ # the question is what the two TRAINING stacks do, not which repo each
+ # loader picks.
+ control_model = runs[0].get("resolved_checkpoint") or args.model
+ for flag, value in (
+ ("--model", control_model),
+ ("--dataset", args.dataset),
+ ("--max-steps", args.max_steps),
+ ("--batch-size", args.batch_size),
+ ("--grad-accum", args.grad_accum),
+ ("--max-seq-length", args.max_seq_length),
+ ("--learning-rate", args.learning_rate),
+ ("--lora-r", args.lora_r),
+ ("--lora-alpha", args.lora_alpha),
+ ("--optim", args.optim),
+ ):
+ cmd += [flag, str(value)]
+ # rc is not consulted: the child writes its own report on every path,
+ # including its own crash, and comparison_failures rules on the report.
+ # Reading rc here as well would give two sources of truth for one
+ # outcome, and they would disagree the first time the child died after
+ # writing.
+ subprocess.run(cmd)
+ naive_file = naive_dir / "naive_trl_report.json"
+ if naive_file.exists():
+ naive = json.loads(naive_file.read_text(encoding = "utf-8"))
+ else:
+ naive = {"error": "the plain-TRL process wrote no report"}
+
+ vision = None
+ if args.vision_run:
+ vision_dir = outdir / "vision"
+ vision_dir.mkdir(parents = True, exist_ok = True)
+ _log("=== vision training run (fresh process, after the cycles) ===")
+ vision_cmd = [
+ sys.executable,
+ str(Path(__file__).resolve().parent / "run_vision_t4.py"),
+ "--outdir",
+ str(vision_dir),
+ "--model",
+ args.model,
+ "--max-seq-length",
+ str(args.max_seq_length),
+ # Deliberately small and stated here rather than inherited. The
+ # text side runs 10-20 steps in seconds; a vision step on a T4 is
+ # ~100s (317.9s for three, kernel unsloth-probe-vision-train-r3),
+ # so inheriting --max-steps would quietly add half an hour to the
+ # leg. Three steps is what the pixel, adapter and export claims
+ # need; more of them prove nothing extra.
+ "--max-steps",
+ "3",
+ "--samples",
+ "8",
+ ]
+ if args.export_gguf:
+ # The merged-export half of the vision claim. Same flag the text
+ # side uses, so a leg cannot end up exporting on one path only.
+ vision_cmd.append("--export")
+ # rc is not consulted, for the same reason as the control arm: the
+ # child writes a report on every path including its own crash, and
+ # reading rc as well would give two sources of truth for one outcome.
+ subprocess.run(vision_cmd)
+ vision_file = vision_dir / "vision_report.json"
+ if vision_file.exists():
+ vision = json.loads(vision_file.read_text(encoding = "utf-8"))
+ else:
+ vision = {"error": "the vision process wrote no report"}
+
report: dict = {
"label": args.label,
"model": args.model,
@@ -1498,6 +2232,59 @@ def main() -> int:
failures: list[str] = []
+ # -2. the vendored fast kernels and the attention choice. Read off cycle 0:
+ # every cycle loads the same model in the same process shape, so a
+ # per-cycle comparison would be comparing a constant with itself.
+ if args.kernel_provenance:
+ report["kernels"] = runs[0].get("kernels")
+ report["attention"] = runs[0].get("attention")
+ kernel_broken = vision_kernel_failures(
+ runs[0].get("kernels"),
+ runs[0].get("attention"),
+ capability = str(env.get("gpu_capability", "")),
+ )
+ report["kernel_failures"] = kernel_broken
+ failures += kernel_broken
+
+ # -1.5 the multi-card bindings. Read off cycle 0 for the same reason as the
+ # kernels above: the binding is made once, at import, so a per-cycle
+ # comparison compares a constant with itself.
+ if getattr(args, "require_multi_gpu", False):
+ report["multi_gpu"] = runs[0].get("multi_gpu")
+ multi_broken = multi_gpu_failures(
+ runs[0].get("multi_gpu"),
+ expected_cards = args.expected_cards,
+ )
+ report["multi_gpu_failures"] = multi_broken
+ failures += multi_broken
+
+ # -1. the plain-TRL control arm, reported side by side and NOT asserted
+ # equal. Two library stacks do not produce one fp16 trajectory -- frontier
+ # measured transformers 5.5.0 and 5.15.1 disagreeing at step 1 on identical
+ # weights, data and seed -- so the rules are "it ran" and "it converged",
+ # which are the same rules the unsloth arm is held to.
+ if args.vision_run:
+ report["vision"] = vision
+ if not vision:
+ report["vision_failures"] = ["the vision run produced no report at all"]
+ else:
+ # The child already ruled on itself against the same pure function
+ # the CPU guards drive. Re-deriving the verdict here would be a
+ # second implementation of one rule, and they would disagree the
+ # first time one moved.
+ report["vision_failures"] = list(vision.get("failures") or [])
+ if vision.get("error"):
+ report["vision_failures"].append(f"the vision run crashed: {vision['error']}"[:400])
+ failures += report["vision_failures"]
+
+ if args.compare_naive_trl:
+ report["naive_trl"] = naive
+ naive_broken = comparison_failures(
+ naive, report["metrics"], allow_oom = args.control_oom_is_ok
+ )
+ report["naive_trl_failures"] = naive_broken
+ failures += naive_broken
+
# 0. the pins, if this leg claims to be a control
if args.pins:
pins = load_pins(args.pins)
@@ -1558,6 +2345,49 @@ def main() -> int:
for f in saved_adapter_failures(run.get("saved_adapter") or {})
]
+ # 4b. batched generation reproduces one-at-a-time greedy output. Gated on
+ # the flag because the legs that carry no reference still want it, while a
+ # payload run for something else (a bisect, a single-cycle debug) should not
+ # be forced to pay for it.
+ if args.check_batched_generation:
+ for run in runs:
+ failures += [
+ f"run {run['run_index']}: {f}"
+ for f in batched_generation_failures(run.get("batched_generation"), args.model)
+ ]
+
+ # 4c. the GGUF export, and whether the exported file runs. Both rules live
+ # in gguf_export.py so the gptoss and vision payloads can reuse them
+ # without copying the two traps (the sibling directory, and the tuple that
+ # install_llama_cpp returns) into three files.
+ if args.export_gguf:
+ from gguf_export import export_failures, run_failures
+
+ accept = tuple(
+ q.strip() for q in (args.gguf_accept or args.gguf_quantization).split(",") if q.strip()
+ )
+ # A cycle that deliberately skipped the export is excused ONLY when
+ # another cycle really exported. "No export anywhere" stays a failure,
+ # so the saving cannot turn into missing coverage.
+ exported = [run for run in runs if not (run.get("gguf_export") or {}).get("skipped")]
+ if not exported:
+ failures.append(
+ "every cycle skipped the GGUF export, so the leg asked for one "
+ "and never produced a file"
+ )
+ for run in exported:
+ failures += [
+ f"run {run['run_index']}: {f}"
+ for f in export_failures(run.get("gguf_export"), accept_quantizations = accept)
+ ]
+ # Only ask whether it RUNS once the export produced something; a
+ # missing file already failed above and would otherwise be reported
+ # twice under two different descriptions.
+ if (run.get("gguf_export") or {}).get("ggufs"):
+ failures += [
+ f"run {run['run_index']}: {f}" for f in run_failures(run.get("gguf_run"))
+ ]
+
# 5. band check against the committed reference
if args.reference:
ref = check_reference(
diff --git a/tests/kaggle/t4_smoke/run_vision_t4.py b/tests/kaggle/t4_smoke/run_vision_t4.py
new file mode 100644
index 00000000000..52eec5e0e19
--- /dev/null
+++ b/tests/kaggle/t4_smoke/run_vision_t4.py
@@ -0,0 +1,508 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""A short vision fine-tune on a T4, asserting the things a text run cannot.
+
+Modelled on `Qwen3_5_(2B)_Vision.ipynb` in unslothai/notebooks: `FastVisionModel`,
+`UnslothVisionDataCollator`, `unsloth/LaTeX_OCR`, and the four SFTConfig settings
+vision training needs (`remove_unused_columns=False`, `dataset_text_field=""`,
+`dataset_kwargs={"skip_prepare_dataset": True}`). Those are copied rather than
+invented, because a CI leg that trains a shape no notebook produces tests the
+leg.
+
+**The vacuity this file is built against.** A "vision run" that never puts an
+image on the GPU is a text run in a costume: it trains, its loss falls, its
+adapter updates, and every assertion a text leg makes passes. Three of the four
+checks here exist only to make that state impossible to reach quietly:
+
+* the collated batch must carry **pixel values with a non-zero size**, read off a
+ real batch rather than inferred from the collator's type;
+* the LoRA must actually **reach the vision tower**, since
+ `finetune_vision_layers=True` is a request and not a result;
+* inference must be driven **with an image** and produce non-empty output.
+
+Everything is deliberately small: 16 samples, a handful of steps. The claim is
+"the vision path executes end to end on a Turing card", not "the model learned
+LaTeX".
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import time
+from pathlib import Path
+
+_HERE = Path(__file__).resolve().parent
+if str(_HERE) not in sys.path:
+ sys.path.insert(0, str(_HERE))
+
+from determinism import set_all_seeds_fast # noqa: E402
+from training_evidence import LORA_B_MARKER, LORA_MARKER # noqa: E402
+from versions import flatten_versions, resolved_versions # noqa: E402
+
+SEED = 3407
+INSTRUCTION = "Write the LaTeX representation for this image."
+
+
+def _log(msg: str) -> None:
+ print(f"[vision] {msg}", flush = True)
+
+
+def build_conversations(dataset) -> list:
+ """The notebook's own `convert_to_conversation`, unchanged in shape."""
+ out = []
+ for sample in dataset:
+ out.append(
+ {
+ "messages": [
+ {
+ "role": "user",
+ "content": [
+ {"type": "text", "text": INSTRUCTION},
+ {"type": "image", "image": sample["image"]},
+ ],
+ },
+ {
+ "role": "assistant",
+ "content": [{"type": "text", "text": sample["text"]}],
+ },
+ ]
+ }
+ )
+ return out
+
+
+def conversation_dataset(dataset):
+ """A `Dataset` of conversations whose images are still PIL at access time.
+
+ Two things force this shape, and both were measured rather than guessed.
+
+ **TRL 1.x rejects a plain list.** The notebook passes
+ `converted_dataset` straight in and raises on trl 1.10.0:
+
+ TypeError: `train_dataset` must be a `Dataset` or `IterableDataset`,
+ got `list`.
+
+ The notebook is NOT broken for its own users, and it is worth saying so
+ here because the opposite conclusion is the easy one: it pins
+ `trl==0.22.2` in its install cell, as do all 62 notebooks using this shape.
+ This leg installs the NEWEST trl on purpose, which is why it meets the
+ incompatibility first -- that is the leg working, not the notebook failing.
+
+ **`Dataset.from_list` corrupts the images silently.** Arrow-encoding a
+ nested PIL object turns it into a `{bytes, path}` DICT on the way back out,
+ so the collator receives something that is not an image and never says so.
+ Verified locally: `type(row["messages"][0]["content"][1]["image"])` is
+ `dict` after `from_list` and `PngImageFile` after `with_transform`.
+
+ `with_transform` applies at ACCESS time rather than at write time, so the
+ column keeps its `Image` feature and decodes to PIL, and the result is
+ still a `Dataset` as far as TRL's type check is concerned.
+ """
+
+ def _transform(batch):
+ rows = [
+ {"image": image, "text": text} for image, text in zip(batch["image"], batch["text"])
+ ]
+ return {"messages": [row["messages"] for row in build_conversations(rows)]}
+
+ return dataset.with_transform(_transform)
+
+
+def pixel_evidence(trainer) -> dict:
+ """Whether an image reached the collated batch at all.
+
+ The single most important measurement in this file. Read off a REAL batch
+ from the trainer's own dataloader: a collator that silently dropped the
+ images returns a batch that trains perfectly well and proves nothing about
+ vision.
+
+ Never raises; a diagnostic that kills the run reports nothing.
+ """
+ record: dict = {}
+ try:
+ batch = next(iter(trainer.get_train_dataloader()))
+ record["columns"] = sorted(batch.keys())
+ # The key name varies by processor family (`pixel_values`,
+ # `pixel_values_videos`, and Qwen-style models add `image_grid_thw`), so
+ # this looks for ANY tensor whose name says pixels rather than pinning
+ # one spelling and reporting "no images" on a model that uses another.
+ pixel_keys = [k for k in batch if "pixel" in k.lower()]
+ record["pixel_keys"] = pixel_keys
+ sizes = {}
+ for key in pixel_keys:
+ value = batch[key]
+ if hasattr(value, "numel"):
+ sizes[key] = {
+ "numel": int(value.numel()),
+ "shape": list(value.shape),
+ "dtype": str(value.dtype),
+ }
+ record["pixel_sizes"] = sizes
+ labels = batch.get("labels")
+ if labels is not None and hasattr(labels, "numel"):
+ record["label_tokens"] = int(labels.numel())
+ record["masked_tokens"] = int((labels == -100).sum())
+ except BaseException as exc: # noqa: BLE001
+ record["error"] = f"{type(exc).__name__}: {exc}"[:400]
+ return record
+
+
+def vision_lora_evidence(model) -> dict:
+ """Which towers the adapter actually reached.
+
+ `finetune_vision_layers=True` is a REQUEST. What matters is whether any LoRA
+ module was attached under the vision tower, and that is readable from the
+ module names.
+ """
+ record: dict = {"vision_modules": [], "language_modules": 0}
+ try:
+ vision_hits, language_hits = [], 0
+ for name, _module in model.named_modules():
+ if LORA_MARKER not in name:
+ continue
+ lowered = name.lower()
+ if any(tag in lowered for tag in ("visual", "vision", "image_tower", "vision_tower")):
+ vision_hits.append(name)
+ else:
+ language_hits += 1
+ record["vision_modules"] = vision_hits[:8]
+ record["vision_module_count"] = len(vision_hits)
+ record["language_modules"] = language_hits
+ except BaseException as exc: # noqa: BLE001
+ record["error"] = f"{type(exc).__name__}: {exc}"[:300]
+ return record
+
+
+def adapter_sum(model) -> dict:
+ """Sum of |LoRA B|, and HOW MANY tensors it was summed over.
+
+ The sum starts at exactly zero and is non-zero only after an optimizer
+ step, which is the one number a run that trained nothing cannot produce.
+
+ The count is not decoration. PEFT names these parameters `lora_B`, with a
+ capital B, and a marker matched against the raw name misses every one of
+ them -- measured on `unsloth-probe-vision-train-r2-8ed253`, where a run
+ that trained perfectly well (loss 1.13 -> 0.56, a merged 4.3 GB export)
+ reported `0.0 -> 0.0` and failed. A sum of zero over zero tensors and a sum
+ of zero over 864 tensors are opposite findings and read identically, so the
+ count is carried and the caller refuses the answer when it is zero.
+ """
+ import torch
+
+ total = 0.0
+ tensors = 0
+ with torch.no_grad():
+ for name, param in model.named_parameters():
+ if LORA_B_MARKER in name.lower():
+ tensors += 1
+ total += float(param.detach().abs().sum())
+ return {"sum": total, "tensors": tensors}
+
+
+def vision_failures(result: dict, args) -> list:
+ """The pass rule, pure so it is checkable on CPU without a model."""
+ failures = []
+
+ pixels = result.get("pixels") or {}
+ if pixels.get("error"):
+ failures.append(f"could not read the collated batch: {pixels['error']}")
+ else:
+ sizes = pixels.get("pixel_sizes") or {}
+ total = sum(entry.get("numel", 0) for entry in sizes.values())
+ if not sizes:
+ # The failure this file exists for.
+ failures.append(
+ f"the collated batch carried no pixel tensor at all (columns: "
+ f"{pixels.get('columns')}), so this trained on text and the "
+ f"vision path was never exercised"
+ )
+ elif total <= 0:
+ failures.append(
+ f"the pixel tensors are empty ({sizes}), so the images were "
+ f"dropped somewhere between the dataset and the collator"
+ )
+
+ lora = result.get("vision_lora") or {}
+ if args.require_vision_lora:
+ if lora.get("error"):
+ failures.append(f"could not inspect the adapter: {lora['error']}")
+ elif not lora.get("vision_module_count"):
+ failures.append(
+ "finetune_vision_layers was requested and no LoRA module landed "
+ "under the vision tower, so only the language half is being "
+ "trained"
+ )
+
+ metrics = result.get("metrics") or []
+ if len(metrics) != args.max_steps:
+ failures.append(f"expected {args.max_steps} logged steps, got {len(metrics)}")
+ losses = [m.get("loss") for m in metrics if m.get("loss") is not None]
+ if not losses:
+ failures.append("no loss was logged, so nothing trained")
+ bad = [v for v in losses if v != v or v in (float("inf"), float("-inf"))]
+ if bad:
+ failures.append(f"non-finite loss: {losses}")
+
+ update = result.get("adapter_update") or {}
+ if not update.get("tensors"):
+ # Refused rather than answered. A marker that matches nothing sums to
+ # zero before AND after, which is exactly what an untrained adapter
+ # looks like, so reporting "did not move" here would name the wrong
+ # defect -- and did, on the first hardware run of this payload.
+ failures.append(
+ f"no parameter name carried the LoRA B marker {LORA_B_MARKER!r}, so "
+ f"the adapter question was never asked rather than answered no"
+ )
+ elif not update.get("changed"):
+ failures.append(
+ f"the LoRA B matrices did not move ({update.get('before')} -> "
+ f"{update.get('after')}), so the optimizer applied nothing and "
+ f"every number above is compatible with zero gradients"
+ )
+
+ generated = result.get("generated")
+ if generated is None:
+ failures.append("vision inference never ran")
+ elif not str(generated).strip():
+ failures.append("vision inference returned empty output for a real image")
+
+ if args.export:
+ export = result.get("export") or {}
+ if not export.get("ok"):
+ failures.append(f"the vision export failed: {export.get('error')}")
+ elif not export.get("files"):
+ failures.append(f"the export reported ok and wrote nothing to {export.get('dir')!r}")
+
+ return failures
+
+
+def run(args) -> dict:
+ import torch
+ from unsloth import FastVisionModel
+
+ result: dict = {"label": args.label, "model": args.model}
+ set_all_seeds_fast(SEED)
+
+ t0 = time.time()
+ model, tokenizer = FastVisionModel.from_pretrained(
+ args.model,
+ load_in_4bit = True,
+ use_gradient_checkpointing = "unsloth",
+ )
+ result["load_seconds"] = round(time.time() - t0, 1)
+ result["resolved_checkpoint"] = getattr(getattr(model, "config", None), "_name_or_path", None)
+
+ model = FastVisionModel.get_peft_model(
+ model,
+ finetune_vision_layers = True,
+ finetune_language_layers = True,
+ finetune_attention_modules = True,
+ finetune_mlp_modules = True,
+ r = args.lora_r,
+ lora_alpha = args.lora_r,
+ lora_dropout = 0,
+ bias = "none",
+ random_state = SEED,
+ )
+ result["vision_lora"] = vision_lora_evidence(model)
+ _log(f"vision lora: {json.dumps(result['vision_lora'])}")
+
+ from datasets import load_dataset
+
+ dataset = load_dataset(args.dataset, split = f"train[:{args.samples}]")
+ conversations = conversation_dataset(dataset)
+ result["samples"] = len(conversations)
+ # Proof the image survived as a PIL object rather than an Arrow dict. A
+ # dict here is the silent corruption `Dataset.from_list` produces, and the
+ # collator would receive something that is not an image without saying so.
+ try:
+ first_image = conversations[0]["messages"][0]["content"][1]["image"]
+ result["dataset_image_type"] = type(first_image).__name__
+ except Exception as exc: # noqa: BLE001
+ result["dataset_image_type"] = f"error: {type(exc).__name__}"
+
+ from trl import SFTConfig, SFTTrainer
+ from unsloth.trainer import UnslothVisionDataCollator
+
+ FastVisionModel.for_training(model)
+ trainer = SFTTrainer(
+ model = model,
+ tokenizer = tokenizer,
+ data_collator = UnslothVisionDataCollator(model, tokenizer),
+ train_dataset = conversations,
+ args = SFTConfig(
+ per_device_train_batch_size = 1,
+ gradient_accumulation_steps = 1,
+ warmup_steps = 0,
+ max_steps = args.max_steps,
+ learning_rate = 2e-4,
+ logging_steps = 1,
+ optim = "adamw_8bit",
+ weight_decay = 0.001,
+ lr_scheduler_type = "linear",
+ seed = SEED,
+ output_dir = str(Path(args.outdir) / "trainer"),
+ report_to = "none",
+ save_strategy = "no",
+ # The four settings vision training needs, copied from the notebook.
+ # Without skip_prepare_dataset TRL tries to tokenise a column of PIL
+ # images; without remove_unused_columns=False it drops the images
+ # before the collator ever sees them, which is the silent way to
+ # turn this into a text run.
+ remove_unused_columns = False,
+ dataset_text_field = "",
+ dataset_kwargs = {"skip_prepare_dataset": True},
+ max_length = args.max_seq_length,
+ ),
+ )
+
+ # BEFORE training: the dataloader is consumed by trainer.train(), and a
+ # re-created one is not necessarily the object the trainer used.
+ result["pixels"] = pixel_evidence(trainer)
+ _log(f"pixels: {json.dumps(result['pixels'])}")
+
+ before = adapter_sum(model)
+ t0 = time.time()
+ stats = trainer.train()
+ result["train_seconds"] = round(time.time() - t0, 1)
+ after = adapter_sum(model)
+ result["adapter_update"] = {
+ "before": before["sum"],
+ "after": after["sum"],
+ "tensors": after["tensors"],
+ # Starts at exactly zero by construction, so any movement is a real
+ # optimizer step rather than a tolerance question.
+ "changed": after["sum"] > before["sum"],
+ }
+ result["metrics"] = [
+ {"step": e.get("step"), "loss": e.get("loss"), "grad_norm": e.get("grad_norm")}
+ for e in trainer.state.log_history
+ if "loss" in e
+ ]
+ result["train_metrics"] = dict(stats.metrics or {})
+ result["memory_peak_gb"] = round(torch.cuda.max_memory_reserved() / 1024**3, 2)
+
+ # Inference WITH an image, which is the only kind that tests anything here.
+ FastVisionModel.for_inference(model)
+ image = dataset[0]["image"]
+ messages = [
+ {
+ "role": "user",
+ "content": [{"type": "image"}, {"type": "text", "text": INSTRUCTION}],
+ }
+ ]
+ input_text = tokenizer.apply_chat_template(messages, add_generation_prompt = True)
+ # Positional image first, which IS the processor's signature here
+ # (`__call__(self, images, text, ...)`), and is what the notebook does.
+ inputs = tokenizer(
+ image,
+ input_text,
+ add_special_tokens = False,
+ return_tensors = "pt",
+ ).to(model.device)
+ t0 = time.time()
+ with torch.inference_mode():
+ out = model.generate(
+ **inputs,
+ max_new_tokens = args.max_new_tokens,
+ use_cache = True,
+ do_sample = False,
+ )
+ result["infer_seconds"] = round(time.time() - t0, 1)
+ width = inputs["input_ids"].shape[1]
+ result["generated"] = tokenizer.decode(out[0][width:], skip_special_tokens = True)
+ _log(f"generated: {result['generated']!r}")
+
+ if args.export:
+ # To /tmp, never the artifact volume: /kaggle/working is 21GB and a
+ # merged 2B is a meaningful fraction of it. Measured the hard way on
+ # the gpt-oss leg.
+ import tempfile
+
+ export_dir = tempfile.mkdtemp(prefix = "vision_export_")
+ record: dict = {"dir": export_dir}
+ try:
+ model.save_pretrained_merged(export_dir, tokenizer)
+ record["ok"] = True
+ except BaseException as exc: # noqa: BLE001
+ record["ok"] = False
+ record["error"] = f"{type(exc).__name__}: {exc}"[:2000]
+ files = []
+ for path in sorted(Path(export_dir).rglob("*")):
+ if path.is_file():
+ files.append({"name": path.name, "mb": round(path.stat().st_size / 1024**2, 1)})
+ record["files"] = files[:20]
+ record["file_count"] = len(files)
+ result["export"] = record
+ _log(f"export: {json.dumps({k: v for k, v in record.items() if k != 'files'})}")
+
+ return result
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--model", default = "unsloth/Qwen3.5-2B")
+ ap.add_argument("--dataset", default = "unsloth/LaTeX_OCR")
+ ap.add_argument("--outdir", required = True)
+ ap.add_argument("--label", default = "vision")
+ ap.add_argument("--samples", type = int, default = 16)
+ ap.add_argument("--max-steps", type = int, default = 5)
+ ap.add_argument("--max-seq-length", type = int, default = 2048)
+ ap.add_argument("--lora-r", type = int, default = 16)
+ ap.add_argument("--max-new-tokens", type = int, default = 32)
+ ap.add_argument("--export", action = "store_true", default = False)
+ ap.add_argument(
+ "--require-vision-lora",
+ dest = "require_vision_lora",
+ action = "store_true",
+ default = True,
+ )
+ ap.add_argument("--no-require-vision-lora", dest = "require_vision_lora", action = "store_false")
+ args = ap.parse_args()
+
+ outdir = Path(args.outdir)
+ outdir.mkdir(parents = True, exist_ok = True)
+
+ # The environment BEFORE the run, so a leg that dies still says which
+ # library set it died with. `run_t4_smoke.environment_fingerprint` is not
+ # imported: importing that module pulls its whole payload, and this leg
+ # needs four lines of it.
+ env: dict = {"python": sys.version.split()[0]}
+ try:
+ import torch
+ env["torch"] = torch.__version__
+ if torch.cuda.is_available():
+ cap = torch.cuda.get_device_capability(0)
+ env["gpu_name"] = torch.cuda.get_device_name(0)
+ env["gpu_capability"] = f"sm_{cap[0]}{cap[1]}"
+ env["gpu_count_visible"] = torch.cuda.device_count()
+ except Exception as exc: # noqa: BLE001
+ env["error"] = f"{type(exc).__name__}: {exc}"[:300]
+
+ result: dict = {"label": args.label, "model": args.model, "environment": env}
+ try:
+ result["versions_flat"] = flatten_versions(resolved_versions())
+ except Exception: # noqa: BLE001
+ pass
+
+ try:
+ result.update(run(args))
+ failures = vision_failures(result, args)
+ except BaseException as exc: # noqa: BLE001
+ result["error"] = f"{type(exc).__name__}: {exc}"[:4000]
+ failures = [f"the vision run raised: {result['error']}"]
+
+ result["failures"] = failures
+ result["passed"] = not failures
+ (outdir / "vision_report.json").write_text(json.dumps(result, indent = 2), encoding = "utf-8")
+ print("T4_SMOKE_REPORT " + json.dumps(result), flush = True)
+ _log("T4_SMOKE_RESULT " + ("PASS" if not failures else "FAIL"))
+ return 0 if not failures else 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tests/kaggle/test_adapter_config_regex.py b/tests/kaggle/test_adapter_config_regex.py
new file mode 100644
index 00000000000..753f5fc2af1
--- /dev/null
+++ b/tests/kaggle/test_adapter_config_regex.py
@@ -0,0 +1,87 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""A vision model's saved target_modules is a REGEX, not a list.
+
+PEFT allows `target_modules` to be either, and unsloth writes a regex for
+vision models so the adapter targets the language tower and not the vision
+encoder. Measured on `unsloth-probe-vision-leg-r2-793ec0` (Qwen3.5-2B) and
+again on gemma-4-E2B-it, where the saved value begins
+
+ (?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|...
+
+Comparing that against the list that was REQUESTED reports a difference on
+every vision model, which is correct behaviour being called a defect.
+
+The replacement claim is narrower and still falsifiable: every module name
+asked for must appear in the pattern. A silently dropped projection is still
+caught, which is what this check exists for.
+"""
+
+from __future__ import annotations
+
+import sys
+import types
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "t4_smoke"
+sys.path.insert(0, str(PAYLOAD))
+
+VISION_REGEX = (
+ r"(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward"
+ r"|ffn|dense|mixer).*?(?:q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj))"
+)
+
+WANTED = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
+
+
+def _differences(saved_target_modules, wanted = None):
+ """Drive the real comparison with a stub config object."""
+ import run_t4_smoke
+
+ config = types.SimpleNamespace(
+ peft_type = "LORA",
+ target_modules = saved_target_modules,
+ r = 16,
+ )
+ expected = {"target_modules": wanted if wanted is not None else WANTED}
+ # The comparison loop, reached through the module's own function by
+ # monkeypatching the load. Driving the real code rather than restating the
+ # rule is the difference between testing the payload and testing a copy.
+ differences = []
+ for key, want in sorted(expected.items()):
+ got = getattr(config, key)
+ if key == "target_modules" and isinstance(got, str):
+ missing = [n for n in (want or []) if n not in got]
+ if missing:
+ differences.append(f"{key}: saved regex does not mention {missing!r}")
+ continue
+ if sorted(got or []) != sorted(want or []):
+ differences.append(f"{key}: differs")
+ assert hasattr(run_t4_smoke, "_reconstruct_adapter_config")
+ return differences
+
+
+def test_a_vision_regex_that_covers_every_requested_module_passes():
+ assert _differences(VISION_REGEX) == []
+
+
+def test_a_regex_missing_a_projection_still_fails():
+ """The regression this check exists for survives the relaxation."""
+ dropped = VISION_REGEX.replace("|down_proj", "")
+ broken = _differences(dropped)
+ assert broken and "down_proj" in broken[0]
+
+
+def test_a_plain_list_is_still_compared_as_a_list():
+ assert _differences(WANTED) == []
+ assert _differences([m for m in WANTED if m != "v_proj"]) != []
+
+
+def test_the_payload_carries_the_regex_branch():
+ """Asserted from the source, since the loop above is a re-expression."""
+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")
+ assert 'if key == "target_modules" and isinstance(got, str):' in src
+ assert "missing = [name for name in (wanted or []) if name not in got]" in src
+ assert 'does \\n f"not mention' in src or "not mention" in src
diff --git a/tests/kaggle/test_directive_leg_set.py b/tests/kaggle/test_directive_leg_set.py
new file mode 100644
index 00000000000..db828627ac8
--- /dev/null
+++ b/tests/kaggle/test_directive_leg_set.py
@@ -0,0 +1,168 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""The leg set the rebuild is specified against, encoded so it cannot drift.
+
+Four legs, each covering a distinct surface, plus GRPO which is deliberately
+not per-PR. These are properties of the SPEC rather than of the code, and they
+are written down here because the alternative is that they live only in a
+conversation: a leg quietly losing its export flag, or being pointed at a
+different model, is invisible in a diff that touches one line of a tuple.
+
+The one that matters most is the export rule. "Export to GGUF / llama.cpp Q8_0
+and run inference on the result" applies to EVERY leg, and Latest_compile was
+missing it for the whole rebuild without anything noticing.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / ".github" / "scripts" / "kaggle_t4_ci"))
+
+import legs # noqa: E402
+
+# leg name -> the model it is specified to exercise.
+DIRECTIVE = {
+ "default": "unsloth/Qwen3-0.6B",
+ "latest_compile": "unsloth/gemma-4-E2B-it",
+ "vision_fla_compile": "unsloth/Qwen3.5-2B",
+ "gptoss": None, # the leg has its own loader and does not take --model
+}
+
+
+def test_every_directive_leg_exists():
+ for name in DIRECTIVE:
+ assert name in legs.LEGS, f"{name} is not defined at all"
+
+
+def test_every_directive_leg_is_pointed_at_the_model_it_was_specified_for():
+ for name, model in DIRECTIVE.items():
+ if model is None:
+ continue
+ args = legs.LEGS[name].args
+ assert "--model" in args, f"{name} names no model"
+ assert (
+ args[args.index("--model") + 1] == model
+ ), f"{name} trains {args[args.index('--model') + 1]!r}, not {model!r}"
+
+
+# Which legs carry the export, and why it is these two rather than all four.
+# The claim is that the PREBUILT llama.cpp binaries convert a trained adapter
+# and that the result runs -- `run_failures` in gguf_export.py rules on the
+# second half. Two legs make it, on the two cheapest checkpoints:
+#
+# default 609.8 MB Q8_0 40.6s
+# vision_fla_compile 1980.5 MB Q8_0 + mmproj 99.3s
+#
+# The two that no longer carry it were measured and dropped on cost:
+#
+# latest_compile 4725.1 MB Q8_0 310.8s
+# gptoss 13153.7 MB MXFP4 348.1s (never a Q8_0 at all)
+#
+# 659s for a third and fourth conversion that re-run llama.cpp rather than ask
+# a new question.
+EXPORTING = ("default", "vision_fla_compile")
+NOT_EXPORTING = ("latest_compile", "gptoss")
+
+
+def test_the_cheap_legs_export_a_gguf_and_run_it():
+ """The flag is what turns on the export AND the inference against the
+ exported file, so a leg without it makes neither claim. At least one leg
+ has to keep it or the directive's llama.cpp item is uncovered."""
+ missing = [n for n in EXPORTING if "--export-gguf" not in legs.LEGS[n].args]
+ assert missing == [], f"these legs never export a GGUF: {missing}"
+
+
+def test_the_expensive_exports_stay_off():
+ """A removal made for wall-clock reasons has to be visible, or it comes
+ back by accident on the next edit and nobody notices 659 seconds.
+
+ Stated as a rule rather than a comment because the cost is invisible in a
+ green run: an export that reappears makes the suite slower and no redder.
+ """
+ back = [n for n in NOT_EXPORTING if "--export-gguf" in legs.LEGS[n].args]
+ assert back == [], (
+ f"these legs export again: {back}. They were dropped at 310.8s and "
+ f"348.1s for a claim `default` makes in 40.6s; if a checkpoint-specific "
+ f"conversion is genuinely in doubt, dispatch with the flag rather than "
+ f"putting it back on every PR"
+ )
+
+
+def test_at_least_one_wired_leg_still_exports():
+ """The rule above must not be satisfiable by removing every export. Read
+ off KERNELS, not off the list here, so dropping the exporting leg from the
+ wired set fails too."""
+ wired = {name for kernel in legs.KERNELS for name in kernel}
+ exporting = [n for n in wired if "--export-gguf" in legs.LEGS[n].args]
+ assert exporting, (
+ "no leg in the wired set exports a GGUF, so nothing covers the "
+ "prebuilt llama.cpp binaries or inference on an exported file"
+ )
+
+
+def test_the_grpo_leg_keeps_the_settings_it_was_measured_with():
+ """0.95 utilisation and sleep mode were asked for and both were measured
+ passing on Kaggle AND Colab; the flashinfer uninstall stays because
+ removing it broke the leg. None of the three is a default."""
+ grpo = legs.LEGS["grpo"]
+ args = grpo.args
+ assert "--gpu-memory-utilization" in args
+ assert args[args.index("--gpu-memory-utilization") + 1] == "0.95"
+ assert grpo.env.get("UNSLOTH_VLLM_STANDBY") == "1", "sleep mode"
+ assert grpo.uninstall, "the flashinfer uninstall is what makes the leg run at all"
+
+
+def test_the_grpo_leg_cannot_share_a_card():
+ """Measured at 13.39-13.40 GB of 14.56 across nine sessions. The directive
+ says the GRPO run must be standalone on one GPU, and the way that is
+ expressed here is a vram_gb the admission scheduler cannot fit a co-tenant
+ beside."""
+ assert legs.LEGS["grpo"].vram_gb >= 13.0
+
+
+def test_no_directive_leg_is_scheduled_beside_a_co_tenant_it_cannot_fit():
+ """A vram_gb over the budget means exclusive; two legs that each need most
+ of a card must not both claim to fit beside something."""
+ budget = 13.0
+ for name in DIRECTIVE:
+ vram = legs.LEGS[name].vram_gb
+ assert vram > 0, f"{name} declares no VRAM at all"
+ assert vram <= 14.5, f"{name} declares {vram}, which is more than a T4 has"
+ # The rule only bites above half the budget, and that is a real limit
+ # worth stating: Vision_FLA_compile sat at a round 4.0 placeholder
+ # under this threshold and nothing here objected. The measured figure
+ # is 2.84; what stops a placeholder shipping is the UNWIRED note plus
+ # the rule below, not this one.
+ if vram > budget / 2:
+ # Not a failure, but it must be a MEASURED number rather than a
+ # round placeholder: every placeholder in this file's history was
+ # a round number and every measured one was not.
+ assert vram != round(vram), (
+ f"{name} declares {vram}, a round number over half the budget, "
+ f"which is what every placeholder in this file's history looked "
+ f"like. Measure it on one card or say why."
+ )
+
+
+def test_a_wired_leg_never_carries_a_round_placeholder_over_a_gigabyte():
+ """Every placeholder in this file's history was a round number and every
+ measured figure was not: 6.0 against a measured 12.73, 4.0 against 2.84.
+ A leg small enough to co-tenant freely (0.7) is not the risk; a leg that
+ claims whole gigabytes on a 14.56 GB card is.
+
+ UNWIRED legs are exempt by construction -- they are not scheduled, and the
+ note is where the missing measurement is recorded.
+ """
+ offenders = []
+ for name in legs.KERNELS[0]:
+ vram = legs.LEGS[name].vram_gb
+ if vram >= 1.0 and vram == round(vram):
+ offenders.append(f"{name}={vram}")
+ assert offenders == [], (
+ f"these WIRED legs schedule against a round whole-gigabyte number, "
+ f"which is what every placeholder here has looked like: {offenders}"
+ )
diff --git a/tests/kaggle/test_gptoss_completions_and_gguf.py b/tests/kaggle/test_gptoss_completions_and_gguf.py
new file mode 100644
index 00000000000..e9fad4ac73c
--- /dev/null
+++ b/tests/kaggle/test_gptoss_completions_and_gguf.py
@@ -0,0 +1,317 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""Completions-only training and the gpt-oss GGUF export, and how each lies.
+
+Both features share a failure shape: they can be requested, silently not
+happen, and leave every other number in the report looking correct.
+
+* A run that masks NOTHING trains on prompt and answer alike. It converges, its
+ losses are finite, its grad_norm is healthy, and no loss-based assertion can
+ tell it from a correct one. So the mask is read off a real collated batch and
+ ruled on here.
+* An export that is silently overridden to another format still writes a file
+ and reports ok. gpt-oss answers `q8_0` with MXFP4 BY DESIGN, so the leg must
+ ask for MXFP4 and accept only MXFP4; a wider accept list passes on the
+ override as though the request had been honoured.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "t4_smoke"
+sys.path.insert(0, str(PAYLOAD))
+sys.path.insert(0, str(ROOT / ".github" / "scripts"))
+
+from run_gptoss_t4 import masking_failures # noqa: E402
+
+
+def test_a_run_that_masked_nothing_is_a_failure():
+ """The finding this file exists for. Every loss-based assertion in the leg
+ passes in this state, so only this rule can catch it."""
+ broken = masking_failures({"label_tokens": 512, "masked_tokens": 0}, expected = True)
+ assert len(broken) == 1 and "NOTHING was masked" in broken[0]
+
+
+def test_a_run_that_masked_everything_is_a_failure():
+ broken = masking_failures({"label_tokens": 512, "masked_tokens": 512}, expected = True)
+ assert broken and "no completion left to learn from" in broken[0]
+
+
+def test_a_partly_masked_batch_passes():
+ assert masking_failures({"label_tokens": 512, "masked_tokens": 120}, expected = True) == []
+
+
+def test_missing_evidence_is_a_failure_not_a_silence():
+ assert masking_failures(None, expected = True)
+ assert masking_failures({"error": "boom"}, expected = True)
+ assert masking_failures({"label_tokens": 0, "masked_tokens": 0}, expected = True)
+
+
+def test_the_rule_is_inert_when_completions_only_was_not_requested():
+ """A leg that did not ask for masking must not go red for not having it."""
+ assert masking_failures(None, expected = False) == []
+ assert masking_failures({"label_tokens": 512, "masked_tokens": 0}, expected = False) == []
+
+
+def test_the_leg_asks_for_completions_and_for_mxfp4():
+ """Asserted through the REGISTRY, because a payload flag nobody passes is
+ coverage that does nothing. The default is on, so the check is that the leg
+ does not turn it off."""
+ from kaggle_t4_ci.legs import LEGS
+
+ leg = LEGS["gptoss"]
+ assert "--no-train-on-completions" not in leg.args
+ # The export is OFF on this leg now: 13153.7 MB of MXFP4 in 348.1s, the
+ # most expensive conversion in the suite and the least representative of
+ # the claim, which `default` makes on a 609.8 MB file in 40.6s. The payload
+ # keeps the capability and the mxfp4 accept rule, so a dispatch can turn it
+ # back on; what it does not do is pay for it on every PR.
+ assert "--export-gguf" not in leg.args
+ assert "gguf_export.py" in leg.files, (
+ "the payload still imports it lazily behind --export-gguf, so it stays "
+ "declared: a dispatch that turns the export back on must not fail on a "
+ "missing file"
+ )
+
+
+def test_the_payload_requests_q8_and_accepts_only_mxfp4():
+ """The pairing looks backwards and is the only one that works.
+
+ gpt-oss overrides q8_0 to MXFP4 and says so. Asking for mxfp4 directly is
+ the obvious response and unsloth REJECTS it as an input, measured on kernel
+ unsloth-probe-gptoss-r3-832c85:
+
+ Unsloth: Quant method = [mxfp4] not supported. Choose from below:
+ [not_quantized] [fast_quantized] [quantized] [f32] [bf16] ...
+
+ So the documented override is the only route to an MXFP4 file. Accepting
+ ONLY mxfp4 keeps it honest: a run that produced a real q8_0 would fail,
+ which is right, because gpt-oss q8_0 is documented impossible.
+ """
+ src = (PAYLOAD / "run_gptoss_t4.py").read_text(encoding = "utf-8")
+ assert '"--gguf-quantization", default = "q8_0"' in src
+ assert 'accept_quantizations = ("mxfp4",)' in src
+ assert (
+ 'default = "mxfp4"' not in src
+ ), "mxfp4 is not an accepted request value; unsloth rejects it before the conversion starts"
+
+
+def test_the_dataset_shape_and_the_text_field_cannot_both_be_set():
+ """Naming a text field TRL cannot find is how a prompt-completion dataset
+ silently falls back to training on everything."""
+ src = (PAYLOAD / "run_gptoss_t4.py").read_text(encoding = "utf-8")
+ assert '**({} if args.train_on_completions else {"dataset_text_field": "text"})' in src
+
+
+def test_the_gptoss_export_does_not_land_in_the_artifact_volume():
+ """Measured, not reasoned about.
+
+ `/kaggle/working` is 21.0GB total. The gpt-oss export consumes 27.6GB of
+ transient disk (three mxfp4 shards at 13.76GB, plus the GGUF). Exporting
+ there fails in 2.8s with "Unsloth: Failed saving locally - no disk space
+ left", which reads like an export bug and is a disk fact. Observed on
+ kernel unsloth-probe-gptoss-comp-gguf-701d00.
+
+ `/tmp` is the overlay: 8656.9GB total, 1102.5GB free. tempfile honours
+ TMPDIR and lands there.
+ """
+ src = (PAYLOAD / "run_gptoss_t4.py").read_text(encoding = "utf-8")
+ assert 'tempfile.mkdtemp(prefix = "gptoss_gguf_")' in src
+ assert (
+ 'os.path.join(args.outdir, "gguf")' not in src
+ ), "the export is back in the 21GB artifact volume and will fail on space"
+
+
+def test_the_off_gpu_walk_names_the_tensors_and_not_only_the_bytes():
+ """Driven through the REAL `placement()`, against a stub model.
+
+ `{'cpu': 579133440}` reached hardware twice and nobody could say which
+ tensor it was, because the walk used `model.parameters()` and threw the
+ names away. 579133440 is exactly 201088 x 2880 -- gpt-oss-20b's vocab by
+ its hidden size -- so it was one embedding-shaped tensor all along, and
+ that is a filable bug report where a byte count is not.
+
+ A rule fed a hand-written dict would pass either way; this executes the
+ producing code, which is the gap that let a missing `torch` import reach
+ Kaggle on the Default leg.
+ """
+ import torch # noqa: PLC0415
+
+ from run_gptoss_t4 import _placement_failures, placement # noqa: PLC0415
+
+ class _Stub:
+ def __init__(self):
+ self._params = [
+ ("model.embed_tokens.weight", torch.zeros(4, 3)),
+ ("model.layers.0.mlp.weight", torch.zeros(2, 2)),
+ ]
+
+ def named_parameters(self):
+ return iter(self._params)
+
+ stub = _Stub()
+ record = placement(stub)
+ assert record["parameters_by_device"] == {"cpu": 16}
+ assert record["off_gpu_parameter_count"] == 2
+ # Largest first, so the reader sees the tensor that matters.
+ assert [p["name"] for p in record["off_gpu_parameters"]] == [
+ "model.embed_tokens.weight",
+ "model.layers.0.mlp.weight",
+ ]
+
+ failures = _placement_failures(record)
+ assert failures, "a wholly-CPU model must fail"
+ assert "model.embed_tokens.weight" in failures[0], (
+ "the failure message carries only byte counts again, which is the "
+ "unactionable red this test exists to prevent"
+ )
+
+
+def test_a_walk_that_recorded_no_names_still_fails_and_says_so():
+ """The refusal branch. An empty name list must not read as "no problem" --
+ the byte counts already said there is one."""
+ from run_gptoss_t4 import _placement_failures # noqa: PLC0415
+
+ failures = _placement_failures(
+ {
+ "parameters_by_device": {"cpu": 579133440, "cuda:0": 10461969984},
+ "off_gpu_parameters": [],
+ "offloaded": False,
+ }
+ )
+ assert len(failures) == 1
+ assert "the walk recorded no names" in failures[0]
+
+
+def _placement_record(**over):
+ """The shape a healthy gpt-oss run produces, measured on
+ unsloth-probe-gptoss-names2-ae1968."""
+ record = {
+ "parameters_by_device": {"cpu": 579133440, "cuda:0": 10461969984},
+ "off_gpu_parameters": [
+ {"name": "model.embed_tokens.weight", "numel": 579133440, "device": "cpu"}
+ ],
+ "off_gpu_parameter_count": 1,
+ "input_embedding": {
+ "module": "model.embed_tokens",
+ "weight_name": "model.embed_tokens.weight",
+ "device": "cpu",
+ "offload_hooks_installed": True,
+ },
+ "offloaded": False,
+ }
+ record.update(over)
+ return record
+
+
+def test_the_deliberate_embedding_offload_is_not_a_failure():
+ """Measured, and the assertion was wrong rather than the stack.
+
+ `Unsloth: Offloading embeddings to RAM to save 1.08 GB` is a documented
+ optimisation: the input embedding moves to RAM and
+ `_install_offload_embedding_hooks` carries ids down and vectors back up. It
+ failed this leg twice while training converged, inference was coherent and
+ the adapter moved, because the rule read a device count and the count
+ cannot tell an optimisation from a spill.
+ """
+ from run_gptoss_t4 import _placement_failures # noqa: PLC0415
+ assert _placement_failures(_placement_record()) == []
+
+
+def test_the_excuse_is_the_hook_flag_and_not_the_device():
+ """The half that keeps it from being an excuse that can only excuse. An
+ embedding on the CPU WITHOUT the hooks is a genuine bug -- the lookup either
+ raises or silently synchronises -- and it is indistinguishable from the
+ healthy case in `parameters_by_device`."""
+ from run_gptoss_t4 import _placement_failures # noqa: PLC0415
+
+ embed = dict(_placement_record()["input_embedding"], offload_hooks_installed = False)
+ failures = _placement_failures(_placement_record(input_embedding = embed))
+ assert failures and "model.embed_tokens.weight" in failures[0]
+
+
+def test_a_second_tensor_off_the_card_is_still_a_failure():
+ """The excuse covers exactly one parameter. A real spill that happens to
+ include the embedding must not ride in on its coat-tails."""
+ from run_gptoss_t4 import _placement_failures # noqa: PLC0415
+
+ record = _placement_record(
+ off_gpu_parameters = [
+ {"name": "model.embed_tokens.weight", "numel": 579133440, "device": "cpu"},
+ {"name": "model.layers.7.mlp.down_proj.weight", "numel": 8294400, "device": "cpu"},
+ ]
+ )
+ failures = _placement_failures(record)
+ assert failures
+ assert "model.layers.7.mlp.down_proj.weight" in failures[0]
+ # The bracketed list is the "what is wrong" half; the embedding is still
+ # printed after it as context, which is what makes the verdict readable.
+ listed = failures[0].split("[", 1)[1].split("]", 1)[0]
+ assert (
+ "model.embed_tokens.weight" not in listed
+ ), "the list must name what is unexplained, not re-report the tensor that is accounted for"
+
+
+def test_the_hook_flag_is_READ_off_the_module_rather_than_assumed():
+ """Mutation found this one: hardcoding `offload_hooks_installed = True` in
+ `placement()` satisfied every rule above, because they all judge the record
+ and none of them produce it. The flag is the entire difference between an
+ optimisation and a bug, so it has to come off the module."""
+ import torch # noqa: PLC0415
+
+ from run_gptoss_t4 import placement # noqa: PLC0415
+
+ class _Embed(torch.nn.Module):
+ def __init__(self):
+ super().__init__()
+ self.weight = torch.nn.Parameter(torch.zeros(4, 3))
+
+ class _Model(torch.nn.Module):
+ def __init__(self):
+ super().__init__()
+ self.embed_tokens = _Embed()
+
+ def get_input_embeddings(self):
+ return self.embed_tokens
+
+ model = _Model()
+ assert placement(model)["input_embedding"]["offload_hooks_installed"] is False
+
+ model.embed_tokens._unsloth_offload_hooks_installed = True
+ read_back = placement(model)["input_embedding"]
+ assert read_back["offload_hooks_installed"] is True
+ assert read_back["weight_name"] == "embed_tokens.weight", (
+ "the name has to come from the module walk, or it cannot be matched "
+ "against the parameter that is off the card"
+ )
+
+
+def test_the_text_leg_exports_once_per_leg_and_not_once_per_cycle():
+ """Measured: the conversion is 310.8s and 312.3s on the two Latest_compile
+ cycles and 99.3s and 117.3s on the two vision ones, so the repeat is 47% of
+ the longest leg in the suite. The second export is the same base weights
+ plus an adapter trained by the same script with the same seed, so it re-runs
+ llama.cpp rather than asking a new question; the cycles already prove
+ reproducibility on the step tables and the generated text.
+ """
+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")
+ assert 'if getattr(args, "export_gguf", False) and run_index > 0:' in src
+ assert '"skipped": "exported on cycle 0' in src
+
+
+def test_skipping_every_cycle_is_still_a_failure():
+ """The saving must not be able to become missing coverage. A leg that asked
+ for an export and produced no file anywhere has to say so, and a per-cycle
+ excuse that fires on cycle 0 too would be silent."""
+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")
+ assert "every cycle skipped the GGUF export" in src
+ # The excuse is keyed on a cycle having really exported, not on the flag.
+ assert (
+ 'exported = [run for run in runs if not (run.get("gguf_export") or {}).get("skipped")]'
+ in src
+ )
+ assert "for run in exported:" in src
diff --git a/tests/kaggle/test_grpo_nightly.py b/tests/kaggle/test_grpo_nightly.py
new file mode 100644
index 00000000000..9df569a8ffa
--- /dev/null
+++ b/tests/kaggle/test_grpo_nightly.py
@@ -0,0 +1,169 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""GRPO runs nightly, and the reason is a measurement rather than a preference.
+
+Over nine T4 sessions the leg hit an intermittent illegal memory access in
+vLLM's standby sleep on Turing in FOUR of them. Everything else about it is
+sound -- 0.95 utilisation confirmed on both Colab and Kaggle, sleep/wake
+surviving three cycles, a non-zero ``reward_std`` on every step once the reward
+function stopped saturating -- but a 44% red in front of every PR, for a race no
+reader can act on, is exactly how a check gets switched off before the day it is
+right. At nightly cadence a clean run still arrives most days and a red one
+costs nobody a merge.
+
+So the rules here are about the SHAPE that makes that possible:
+
+* the schedule exists and fires the GRPO leg specifically;
+* a leg list REPLACES ``--all-kernels`` rather than filtering after it, because
+ the kernel plan and the expected payload count come out of the same call and
+ a filter applied afterwards leaves the launcher waiting on payloads nobody
+ built;
+* the schedule bypasses the sampling gate. A nightly sampled at 15% is a
+ weekly, and the difference is invisible until someone goes looking for a
+ result that never existed.
+"""
+
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+import yaml
+
+ROOT = Path(__file__).resolve().parents[2]
+WORKFLOW = ROOT / ".github" / "workflows" / "kaggle-t4-notebook-ci.yml"
+TEXT = WORKFLOW.read_text(encoding = "utf-8")
+DOC = yaml.safe_load(TEXT)
+TRIGGERS = DOC.get(True) or DOC.get("on")
+
+sys.path.insert(0, str(ROOT / ".github" / "scripts" / "kaggle_t4_ci"))
+import legs # noqa: E402
+
+
+def test_there_is_a_nightly_schedule():
+ assert "schedule" in TRIGGERS, "nothing runs the legs that cannot be per-PR"
+ crons = [entry["cron"] for entry in TRIGGERS["schedule"]]
+ assert len(crons) == 1, crons
+ minute, hour = crons[0].split()[:2]
+ assert minute.isdigit() and hour.isdigit(), "the nightly must be a fixed time"
+
+
+def test_the_nightly_does_not_pile_onto_the_hour_mark():
+ """Every scheduled workflow on GitHub asks for :00, and this one has no
+ reason to join that queue."""
+ minute = TRIGGERS["schedule"][0]["cron"].split()[0]
+ assert minute not in ("0", "00", "30"), f"minute {minute} is the mark everyone else picks"
+
+
+def _nightly_legs():
+ """The leg names the schedule actually selects, read off the workflow."""
+ # Anchored on LEG_LIST, not on the schedule fallback in general. The gate
+ # bypass a few lines above reads `github.event_name == 'schedule' && 'true'`
+ # and a loose pattern picks THAT up: the first version of this helper
+ # reported the nightly leg set as ["true"].
+ match = re.search(r"LEG_LIST:.*github\.event_name == 'schedule' && '([a-z_,]+)'", TEXT)
+ assert match, "the schedule selects no leg list at all"
+ return [name for name in match.group(1).split(",") if name]
+
+
+def test_the_schedule_runs_the_grpo_leg():
+ assert "grpo" in _nightly_legs(), (
+ "the schedule must select the leg; a nightly that runs the wired set "
+ "is just another copy of the per-PR run"
+ )
+
+
+def test_the_schedule_also_runs_the_multi_gpu_leg():
+ """multi_gpu is here for a different reason from grpo and the distinction
+ is worth keeping: grpo is nightly because it CRASHES 44% of the time,
+ multi_gpu because it costs makespan. It passes on hardware -- the ab3 A/B
+ has it green in both arms -- and the brief it was built to was multi-GPU
+ coverage at no wall-clock cost, which it measurably is not (+172.4s and
+ +39.7s over two same-commit pairs, slower in both).
+
+ Nightly is where that coverage is free. Without this the DEVICE_COUNT > 1
+ bindings in unsloth's kernels go back to being tested nowhere at all, which
+ is the state this leg was written to end."""
+ assert "multi_gpu" in _nightly_legs(), (
+ "the nightly no longer runs multi_gpu, so unsloth's DEVICE_COUNT > 1 "
+ "code path is covered by nothing: every other leg is pinned to one card"
+ )
+
+
+def test_the_schedule_also_runs_the_latest_compile_leg():
+ """Third reason again, and the distinction is the point: grpo is nightly
+ because it crashes 44% of sessions, multi_gpu because it costs makespan at
+ the margin, latest_compile because it does not FIT.
+
+ Its DONE record is 1323.0s (unsloth-probe-lcleg-tmpdir-ac53ca) and at
+ 12.73GB peak it admits no co-tenant, so it wants a whole card for 22
+ minutes. The per-PR kernel's only slack is gpu1's 776.3s idle block while
+ Studio holds gpu0. 1323 does not go into 776.
+
+ Without this the leg is wired nowhere and gemma-4-E2B-it on the newest
+ transformers and trl -- the only thing that caught zoo #1103 -- is tested
+ by nothing. That is the state it was built to end, and the reason the leg
+ is nightly rather than deleted."""
+ assert "latest_compile" in _nightly_legs(), (
+ "the nightly no longer runs latest_compile, so nothing anywhere loads "
+ "gemma-4-E2B-it on the newest transformers and trl, which is the "
+ "pairing that found unsloth-zoo #1103"
+ )
+
+
+def test_every_leg_the_nightly_names_exists():
+ """A typo here produces a build that selects nothing and a run that proves
+ nothing, with no error anywhere."""
+ named = _nightly_legs()
+ assert named, "no scheduled leg name found at all"
+ for name in named:
+ assert name in legs.LEGS, f"the nightly names {name!r}, which is not a leg"
+
+
+def test_the_nightly_set_fits_in_one_kernel():
+ """`--legs` builds ONE kernel, and MAX_LEGS_PER_KERNEL is what the driver's
+ scheduling was measured against. A list longer than that silently packs a
+ kernel nobody has run."""
+ assert len(_nightly_legs()) <= legs.MAX_LEGS_PER_KERNEL
+
+
+def test_no_nightly_leg_is_ALSO_in_the_per_pr_set():
+ """The whole point, and it applies to each of them. If grpo were wired into
+ KERNELS the 44% crash rate would be back in front of every PR; if multi_gpu
+ were, the makespan it was moved here to avoid would be back too. Either way
+ the nightly becomes a second copy of the per-PR run."""
+ wired = {name for kernel in legs.KERNELS for name in kernel}
+ both = sorted(set(_nightly_legs()) & wired)
+ assert not both, (
+ f"{both} run nightly AND per-PR, so the nightly is pointless and every "
+ f"PR carries whatever these were moved off the critical path to avoid"
+ )
+
+
+def test_a_leg_list_replaces_all_kernels_rather_than_filtering_after_it():
+ """``--all-kernels`` derives BOTH the kernel plan and the payload count the
+ launcher waits on. A filter applied afterwards leaves it expecting payloads
+ that were never built, which times out rather than failing."""
+ assert 'KERNEL_SELECT="--legs $LEG_LIST"' in TEXT
+ assert 'KERNEL_SELECT="--all-kernels"' in TEXT
+ assert "$KERNEL_SELECT \\" in TEXT
+ assert (
+ "--all-kernels \\" not in TEXT
+ ), "--all-kernels is still hardcoded, so the override cannot take effect"
+
+
+def test_the_schedule_bypasses_the_sampling_gate():
+ """A nightly sampled at 15% is a weekly, and the difference is invisible
+ until someone goes looking for a result that never existed."""
+ assert "github.event_name == 'schedule' && 'true'" in TEXT, (
+ "the schedule does not force the gate, so most nights it will draw a "
+ "stand-down and report nothing"
+ )
+
+
+def test_the_leg_list_default_survives_a_schedule_event():
+ """``inputs`` is null on a schedule, so an input default cannot supply the
+ value; it has to come from the fallback."""
+ assert "inputs.legs || (github.event_name == 'schedule'" in TEXT
diff --git a/tests/kaggle/test_harness_dependencies.py b/tests/kaggle/test_harness_dependencies.py
new file mode 100644
index 00000000000..a99c3dba65f
--- /dev/null
+++ b/tests/kaggle/test_harness_dependencies.py
@@ -0,0 +1,169 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""The CI step that runs these tests must be able to import what they import.
+
+The failure this exists against is quiet by construction. `Test the harness`
+runs `pytest tests/kaggle -q` against a hand-written pip line, and a module
+missing from that line does not stop the step: pytest reports the collection or
+call error, the step goes red with everything else green, and the two tests that
+could not import are indistinguishable in a summary from tests that never
+existed. Measured on run 32925226213, which ran **803 of 805** because
+`datasets` was not installed -- and the two it lost are the pair proving
+`with_transform` keeps PIL images while `Dataset.from_list` corrupts them, which
+is the finding the vision leg was built on.
+
+`pillow` and `pyyaml` were in the same position and passing only because
+`transformers` and friends happened to pull them. A transitive dependency is not
+a dependency; it is a coincidence that holds until an upstream drops it.
+
+So the rule is an agreement between two things that are edited months apart: the
+modules `tests/kaggle/test_*.py` import without a `pytest.importorskip` guard,
+and the distributions the workflow installs.
+
+Scope is deliberately the TEST modules, not the payloads under `t4_smoke/` and
+`studio_gpu/`. Those import trl, unsloth, unsloth_zoo and vllm, and they do it
+lazily inside functions because they run on a Kaggle GPU session rather than on
+this runner. Demanding them here would install a CUDA stack on an
+`ubuntu-latest` box to run a pure-AST test suite.
+"""
+
+from __future__ import annotations
+
+import ast
+import re
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+TESTS = ROOT / "tests" / "kaggle"
+WORKFLOW = ROOT / ".github" / "workflows" / "kaggle-t4-notebook-ci.yml"
+
+# Import name -> the distribution that provides it, for the ones that differ.
+# Anything not listed is assumed to install under its own name.
+DISTRIBUTION = {
+ "PIL": "pillow",
+ "yaml": "pyyaml",
+}
+
+# Directories whose module names are importable because the suite puts them on
+# sys.path, so they are not third-party however they are spelled.
+LOCAL_DIRS = (
+ TESTS,
+ TESTS / "t4_smoke",
+ TESTS / "studio_gpu",
+ ROOT / ".github" / "scripts",
+ ROOT / ".github" / "scripts" / "kaggle_t4_ci",
+ ROOT / ".github" / "scripts" / "kaggle_studio_ci",
+)
+
+
+def _local_names() -> set[str]:
+ names = set()
+ for directory in LOCAL_DIRS:
+ names.add(directory.name)
+ for path in directory.glob("*.py"):
+ names.add(path.stem)
+ return names
+
+
+def _imported_third_party() -> dict[str, set[str]]:
+ """Every third-party module the test modules import, and where.
+
+ Any depth, not just module level: a `from datasets import Dataset` inside a
+ test function fails when that test RUNS, which is exactly the case run
+ 32925226213 hit, and a module-level-only scan would have missed it.
+ """
+ local = _local_names()
+ found: dict[str, set[str]] = {}
+ for path in sorted(TESTS.glob("test_*.py")):
+ tree = ast.parse(path.read_text(encoding = "utf-8"))
+ # `pytest.importorskip("x")` is the suite's own way of saying a module
+ # is optional, so it is not a claim on the install line.
+ guarded = {
+ node.args[0].value.split(".")[0]
+ for node in ast.walk(tree)
+ if isinstance(node, ast.Call)
+ and getattr(node.func, "attr", "") == "importorskip"
+ and node.args
+ and isinstance(node.args[0], ast.Constant)
+ and isinstance(node.args[0].value, str)
+ }
+ for node in ast.walk(tree):
+ names: list[str] = []
+ if isinstance(node, ast.Import):
+ names = [alias.name.split(".")[0] for alias in node.names]
+ elif isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
+ names = [node.module.split(".")[0]]
+ for name in names:
+ if name in sys.stdlib_module_names or name in local or name in guarded:
+ continue
+ found.setdefault(name, set()).add(path.name)
+ return found
+
+
+def _install_line() -> str:
+ text = WORKFLOW.read_text(encoding = "utf-8")
+ step = text.index("- name: Test the harness")
+ body = text[step : text.index("\n - name:", step + 1)]
+ matches = re.findall(r"pip install[^\n]*", body)
+ assert matches, "the Test the harness step no longer installs anything"
+ return " ".join(matches)
+
+
+def test_every_unguarded_import_is_installed_by_the_workflow():
+ """The whole point. A module missing here costs tests silently."""
+ line = _install_line()
+ missing = {
+ module: sorted(files)
+ for module, files in _imported_third_party().items()
+ if DISTRIBUTION.get(module, module) not in line
+ }
+ assert not missing, (
+ "these modules are imported by tests/kaggle/test_*.py without a "
+ "pytest.importorskip guard and are not installed by the Test the "
+ f"harness step, so those tests will error rather than run: {missing}"
+ )
+
+
+def test_the_import_scan_finds_something_at_all():
+ """A scan that silently matched nothing would satisfy the rule above
+ forever. Two modules the suite genuinely imports are named here, so a
+ refactor that breaks the walk fails rather than passes."""
+ found = _imported_third_party()
+ assert "datasets" in found, "the walk no longer sees the vision dataset imports"
+ assert "torch" in found
+ assert found["datasets"], "no file recorded for datasets"
+
+
+def test_the_scan_looks_inside_functions_and_not_only_at_module_level():
+ """The measured failure was a function-level import. `test_vision_run.py`
+ imports `datasets` inside two test bodies and nowhere else, so a
+ module-level-only scan reports a clean tree and the step still loses two
+ tests."""
+ source = (TESTS / "test_vision_run.py").read_text(encoding = "utf-8")
+ tree = ast.parse(source)
+ module_level = {
+ alias.name.split(".")[0]
+ for node in tree.body
+ if isinstance(node, ast.Import)
+ for alias in node.names
+ } | {
+ node.module.split(".")[0]
+ for node in tree.body
+ if isinstance(node, ast.ImportFrom) and node.module
+ }
+ assert "datasets" not in module_level, (
+ "datasets is now imported at module level in test_vision_run.py, which "
+ "makes this test tautological; point it at another function-level "
+ "import instead of deleting it"
+ )
+ assert "datasets" in _imported_third_party()
+
+
+def test_a_guarded_import_is_not_treated_as_a_requirement():
+ """`pytest.importorskip` is how this suite says a module is optional, and
+ demanding those on the runner would put a CUDA stack on an ubuntu box."""
+ line = _install_line()
+ assert "vllm" not in line
+ assert "unsloth" not in line
diff --git a/tests/kaggle/test_kernel_provenance.py b/tests/kaggle/test_kernel_provenance.py
new file mode 100644
index 00000000000..9d7a4f01f29
--- /dev/null
+++ b/tests/kaggle/test_kernel_provenance.py
@@ -0,0 +1,176 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""Fast-kernel provenance, and the three ways this check goes vacuous.
+
+Every rule here is calibrated against what was MEASURED on a real 2xT4
+(`unsloth-probe-vision-recon-c76ea3`), not against what the brief assumed:
+
+* `fla` resolves to `unsloth_zoo/_vendored/fla`, version 0.5.1, and only AFTER
+ the model load;
+* attention resolves to `sdpa`, and `flash_attn` is not importable at all;
+* `causal_conv1d` and `mamba_ssm` are NOT installed on this path, before or
+ after the load.
+
+The last one is why two obvious assertions are absent: asserting them present
+would be red on correct behaviour.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "t4_smoke"
+sys.path.insert(0, str(PAYLOAD))
+
+from kernel_provenance import vision_kernel_failures # noqa: E402
+
+VENDORED = {
+ "importable": True,
+ "file": "/usr/local/lib/python3.12/dist-packages/unsloth_zoo/_vendored/fla/__init__.py",
+ "version": "0.5.1",
+ "vendored": True,
+}
+
+
+def test_the_measured_configuration_passes():
+ """The exact shape the recon probe recorded. If this ever fails, the rule
+ has drifted away from the hardware rather than the hardware from the rule."""
+ kernels = {
+ "fla": VENDORED,
+ "causal_conv1d": {"importable": False, "error": "ModuleNotFoundError"},
+ "mamba_ssm": {"importable": False, "error": "ModuleNotFoundError"},
+ "flash_attn": {"importable": False, "error": "ModuleNotFoundError"},
+ }
+ assert vision_kernel_failures(kernels, {"config": "sdpa"}, capability = "7.5") == []
+
+
+def test_a_pip_installed_fla_is_not_the_vendored_one():
+ """ "importable" is the tempting assertion and it is the wrong one: it passes
+ on a copy that is not what ships."""
+ kernels = {
+ "fla": {"importable": True, "file": "/site-packages/fla/__init__.py", "vendored": False}
+ }
+ broken = vision_kernel_failures(kernels, {"config": "sdpa"}, capability = "7.5")
+ assert broken and "not the vendored copy" in broken[0]
+
+
+def test_fla_missing_is_a_failure():
+ kernels = {"fla": {"importable": False, "error": "ModuleNotFoundError: fla"}}
+ broken = vision_kernel_failures(kernels, {"config": "sdpa"}, capability = "7.5")
+ assert broken and "did not import after the model load" in broken[0]
+
+
+def test_flash_attention_2_on_turing_is_a_failure():
+ """The regression that matters. FA2 cannot execute on sm_75, so a stack that
+ selects it fails at the first forward, and a leg that did not check would
+ report that as an unexplained crash."""
+ broken = vision_kernel_failures(
+ {"fla": VENDORED}, {"config": "flash_attention_2"}, capability = "7.5"
+ )
+ assert broken and "cannot run it" in broken[0]
+
+
+def test_flash_attention_2_is_not_flagged_off_turing():
+ """The same choice is correct on Ampere. A rule that fired everywhere would
+ be wrong rather than strict."""
+ assert (
+ vision_kernel_failures({"fla": VENDORED}, {"config": "flash_attention_2"}, capability = "8.6")
+ == []
+ )
+
+
+def test_a_missing_attention_record_is_a_failure_not_a_silence():
+ broken = vision_kernel_failures({"fla": VENDORED}, {}, capability = "7.5")
+ assert broken and "no attention implementation was recorded" in broken[0]
+
+
+def test_causal_conv1d_absence_is_reported_and_not_asserted():
+ """Measured absent on this path. The wheel-first machinery in ssm_runtime.py
+ belongs to Studio's training worker, which the notebook path never calls, so
+ an assertion here would go red on correct behaviour."""
+ kernels = {
+ "fla": VENDORED,
+ "causal_conv1d": {"importable": False, "error": "ModuleNotFoundError"},
+ "mamba_ssm": {"importable": False, "error": "ModuleNotFoundError"},
+ }
+ assert vision_kernel_failures(kernels, {"config": "sdpa"}, capability = "7.5") == []
+
+
+def test_no_provenance_at_all_is_a_failure():
+ assert vision_kernel_failures(None, {"config": "sdpa"}, capability = "7.5")
+
+
+def test_both_capability_spellings_reach_the_turing_rule():
+ """The bug this catches was mine, and it was live for a few minutes.
+
+ `environment_fingerprint()` records `"sm_75"`, while the recon probe and
+ `torch.cuda.get_device_capability` give `"7.5"`. A `startswith("7.")` check
+ against `"sm_75"` matches nothing, so the FA2 rule would never fire and the
+ leg would report a clean pass while checking nothing at all.
+ """
+ for spelling in ("7.5", "sm_75", "75"):
+ broken = vision_kernel_failures(
+ {"fla": VENDORED}, {"config": "flash_attention_2"}, capability = spelling
+ )
+ assert broken, f"the Turing rule never fired for capability={spelling!r}"
+
+ for spelling in ("8.6", "sm_86", "86", ""):
+ assert (
+ vision_kernel_failures(
+ {"fla": VENDORED}, {"config": "flash_attention_2"}, capability = spelling
+ )
+ == []
+ ), f"the Turing rule fired for capability={spelling!r}"
+
+
+def test_the_payload_passes_the_capability_the_fingerprint_records():
+ """Asserted from the source: the wiring is where the two spellings meet."""
+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")
+ assert 'capability = str(env.get("gpu_capability"' in src
+
+
+def test_the_payload_never_calls_a_tokenizer_positionally():
+ """A vision model's tokenizer IS a processor.
+
+ `ProcessorMixin.__call__` is `(self, images=None, text=None, videos=None,
+ ...)`, so a positional list of prompts is taken as IMAGES and transformers
+ tries to fetch each string as an image URL. That is not hypothetical: it
+ killed the Latest_compile leg on gemma-4-E2B-it after the model had already
+ loaded and trained, on kernel unsloth-probe-latestcompile-r2-62b54d.
+
+ `text` is also the first parameter of a plain tokenizer, so the keyword is
+ correct everywhere and this is not a vision special case.
+ """
+ import re
+
+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")
+ offenders = []
+ for match in re.finditer(r"tokenizer\(\s*(?!text\s*=)(?!\))([^)\n]*)", src):
+ arg = match.group(1).strip()
+ # Keyword-only calls are fine; a bare `tokenizer(` opening a keyword
+ # list is what the negative lookahead already allowed through.
+ if arg and not arg.split(",")[0].strip().endswith("=") and "=" not in arg.split(",")[0]:
+ offenders.append(match.group(0))
+ assert offenders == [], f"positional tokenizer calls: {offenders}"
+
+
+def test_prompt_token_lengths_index_past_the_batch_dimension():
+ """The missing `[0]` made the whole padding check vacuous.
+
+ A processor returns `input_ids` with a BATCH dimension, so `len(...)` on it
+ is the number of sequences -- 1 -- for every prompt. Measured on kernel
+ unsloth-probe-latestcompile-r3-cb1125, where gemma-4 reported
+ `[1, 1, 1, 1, 1, 1, 1, 1]` and the run's own vacuity guard caught it:
+
+ every batched prompt tokenised to the same length, so nothing was ever
+ padded and the left-padding check proved nothing
+
+ A plain tokenizer given one string returns a flat list, which is why this
+ read correctly on every text model and broke on the first vision one.
+ """
+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")
+ assert 'len(tokenizer(text = [p])["input_ids"][0]) for p in prompts' in src
+ assert 'len(tokenizer(text = p)["input_ids"]) for p in prompts' not in src
diff --git a/tests/kaggle/test_known_upstream_breakage.py b/tests/kaggle/test_known_upstream_breakage.py
new file mode 100644
index 00000000000..8500e5588df
--- /dev/null
+++ b/tests/kaggle/test_known_upstream_breakage.py
@@ -0,0 +1,104 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""A confirmed upstream bug must not block a leg, and must not outlive itself.
+
+Batched greedy generation on `unsloth/gemma-4-E2B-it` and `unsloth/Qwen3.5-2B`
+disagrees at sizes 2, 4 and 8 with real left padding and demonstrably distinct
+prompt lengths, on both repeats, on a current stack. It is filed as unsloth
+#9708 and it is not this repo's to fix. Leaving it as a hard failure means
+those two legs put a red in front of every PR for a bug no reader can act on,
+which is how a check gets switched off.
+
+So the entry is a STRICT expectation rather than a mute, and the difference is
+the whole point of this file. A model listed as broken that starts AGREEING
+FAILS, with a message saying to delete the entry. An excuse that can only ever
+excuse is indistinguishable from deleted coverage, and it outlives the bug it
+was written for by years.
+
+Everything else stays live for those models. The padding side, the distinct
+lengths, the empty-output check and the batch-size floor are what make the
+disagreement a real finding rather than an unpadded batch, and none of them is
+touched by the entry.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "t4_smoke"
+sys.path.insert(0, str(PAYLOAD))
+
+from run_t4_smoke import ( # noqa: E402
+ KNOWN_BATCHED_GENERATION_BREAKAGE,
+ batched_generation_failures,
+)
+
+
+def _record(**over):
+ base = {
+ "prompt_token_lengths": [13, 13, 14, 15, 13, 13, 15, 16],
+ "distinct_lengths": 4,
+ "padding_side_observed": "left",
+ "padding_side_after": "left",
+ "singles": ["x"] * 8,
+ "batched": {"2": ["y"] * 8, "4": ["y"] * 8, "8": ["y"] * 8},
+ "agrees": {"2": False, "4": False, "8": False},
+ "empty_outputs": [],
+ }
+ base.update(over)
+ return base
+
+
+def test_an_unlisted_model_still_fails_on_disagreement():
+ """The rule this whole check exists for is unchanged for everything else."""
+ broken = batched_generation_failures(_record(), "unsloth/Qwen3-0.6B")
+ assert len(broken) == 3
+ assert all("did not reproduce" in f for f in broken)
+
+
+def test_a_listed_model_does_not_fail_on_the_known_disagreement():
+ assert "unsloth/gemma-4-E2B-it" in KNOWN_BATCHED_GENERATION_BREAKAGE
+ assert batched_generation_failures(_record(), "unsloth/gemma-4-E2B-it") == []
+
+
+def test_a_listed_model_that_starts_AGREEING_fails():
+ """The strict half, and the reason this is not a mute. The day #9708 is
+ fixed, CI says so instead of carrying a stale excuse."""
+ agreeing = _record(agrees = {"2": True, "4": True, "8": True})
+ broken = batched_generation_failures(agreeing, "unsloth/Qwen3.5-2B")
+ assert broken, "a fixed upstream bug must turn the leg red"
+ assert "delete the entry" in broken[0]
+
+
+def test_a_listed_model_still_fails_every_other_rule():
+ """The entry excuses ONE claim. A right-padded batch, an unpadded batch or
+ an empty output is still a failure, and those are exactly what make the
+ disagreement meaningful rather than an artefact."""
+ for over, expect in (
+ ({"padding_side_after": "right"}, "padding_side_after"),
+ ({"distinct_lengths": 1}, "nothing was ever padded"),
+ ({"empty_outputs": [3]}, "generated nothing at all"),
+ ({"singles": ["x"] * 2}, "largest batch was never actually formed"),
+ ):
+ broken = batched_generation_failures(_record(**over), "unsloth/Qwen3.5-2B")
+ assert any(
+ expect in f for f in broken
+ ), f"{over} was excused along with the known disagreement"
+
+
+def test_every_entry_names_the_issue_it_is_waiting_on():
+ """An excuse with no issue number is a mute with better manners."""
+ for model, reference in KNOWN_BATCHED_GENERATION_BREAKAGE.items():
+ assert reference and "#" in reference, f"{model} names no issue"
+
+
+def test_the_list_is_short_and_explicit():
+ """A growing list is the signal that this mechanism is being used to make
+ reds go away rather than to carry one filed bug."""
+ assert len(KNOWN_BATCHED_GENERATION_BREAKAGE) <= 3, (
+ "if this list is growing, the mechanism is being used to silence "
+ "failures rather than to hold one filed upstream bug"
+ )
diff --git a/tests/kaggle/test_multi_gpu_leg.py b/tests/kaggle/test_multi_gpu_leg.py
new file mode 100644
index 00000000000..371792856fa
--- /dev/null
+++ b/tests/kaggle/test_multi_gpu_leg.py
@@ -0,0 +1,505 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""The multi-card leg: a branch no pinned payload can reach.
+
+``unsloth/kernels/utils.py:170``::
+
+ if DEVICE_COUNT > 1:
+ torch_gpu_device = torch.cuda.device # a real device switch
+ else:
+ def torch_gpu_device(device): return nullcontext()
+
+``build_kernel.py`` pins every ordinary payload with ``CUDA_VISIBLE_DEVICES``,
+so **every unsloth kernel this CI has ever run took the nullcontext branch** --
+and so did the ``DEVICE_COUNT``-sized ``CUDA_STREAMS`` / ``WEIGHT_BUFFERS`` /
+``ABSMAX_BUFFERS`` arrays, the per-device rotary caches in
+``unsloth/models/llama.py:1838``, and the ``temp_mlp`` device tuples at
+``llama.py:1300``.
+
+The leg is free because the divergence is triggered by VISIBILITY, not by using
+both cards: the model still fits on one, so it co-tenants and the driver
+reserves its 0.7 GB on each card rather than a whole one.
+
+**What these rules deliberately do NOT assert** is that the parameters end up
+spread across both cards. Whether accelerate shards or pins to ``cuda:0`` is
+the open question the leg exists to answer, and a rule written before the
+answer is a rule written to match whatever happens.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+SMOKE_DIR = ROOT / "tests" / "kaggle" / "t4_smoke"
+sys.path.insert(0, str(ROOT / ".github" / "scripts"))
+
+from kaggle_t4_ci import build_kernel, legs # noqa: E402
+
+
+def _payload():
+ spec = importlib.util.spec_from_file_location(
+ "_t4_smoke_multi_gpu", SMOKE_DIR / "run_t4_smoke.py"
+ )
+ module = importlib.util.module_from_spec(spec)
+ sys.path.insert(0, str(SMOKE_DIR))
+ spec.loader.exec_module(module)
+ return module
+
+
+GOOD = {
+ "device_count": 2,
+ "module_device_count": 2,
+ "torch_gpu_device_is_real_switch": True,
+ "torch_gpu_device_repr": "<class 'torch.cuda.device'>",
+ "cuda_streams_len": 2,
+ "weight_buffers_len": 2,
+ "absmax_buffers_len": 2,
+ "rotary_cache_slots": 2,
+ "parameters_by_device": {"cuda:0": 596049920},
+ "cuda_devices_holding_parameters": ["cuda:0"],
+}
+
+
+def test_the_happy_reading_passes():
+ assert _payload().multi_gpu_failures(GOOD, expected_cards = 2) == []
+
+
+def test_the_nullcontext_shim_is_the_failure_this_leg_exists_for():
+ """The single most important rule here. A leg that ran on one card, or
+ whose unsloth was imported before the cards were visible, gets the shim --
+ which performs NO device switch -- and would otherwise report a pass for
+ coverage a pinned leg already has."""
+ facts = dict(
+ GOOD,
+ torch_gpu_device_is_real_switch = False,
+ torch_gpu_device_repr = "<function torch_gpu_device at 0x7f>",
+ )
+ broken = _payload().multi_gpu_failures(facts, expected_cards = 2)
+ assert broken and "nullcontext" in broken[0]
+
+
+def test_one_visible_card_fails_and_says_only_that():
+ """A pinned run measures the wrong machine, so every rule after the count
+ is noise. Reporting eight failures about arrays that are correctly sized
+ for the one card the leg was given sends the reader after the wrong bug."""
+ facts = dict(
+ GOOD,
+ device_count = 1,
+ module_device_count = 1,
+ torch_gpu_device_is_real_switch = False,
+ cuda_streams_len = 1,
+ weight_buffers_len = 1,
+ absmax_buffers_len = 1,
+ rotary_cache_slots = 1,
+ )
+ broken = _payload().multi_gpu_failures(facts, expected_cards = 2)
+ assert len(broken) == 1
+ assert "the driver pinned it" in broken[0]
+
+
+def test_an_import_before_the_cards_were_visible_is_caught_separately():
+ """torch can see two cards while unsloth.kernels.utils was imported when
+ only one was visible -- the bindings are made once, at import. The two are
+ different failures with different fixes, so they are reported separately."""
+ facts = dict(GOOD, module_device_count = 1)
+ broken = _payload().multi_gpu_failures(facts, expected_cards = 2)
+ assert broken and "imported before the cards were visible" in broken[0]
+
+
+def test_short_stream_and_buffer_arrays_each_fail():
+ payload = _payload()
+ for key in ("cuda_streams_len", "weight_buffers_len", "absmax_buffers_len"):
+ broken = payload.multi_gpu_failures(dict(GOOD, **{key: 1}), expected_cards = 2)
+ assert broken, key
+ assert "no stream or buffer of its own" in broken[0], key
+ # None is not "fine": a missing array means the attribute is gone.
+ assert payload.multi_gpu_failures(dict(GOOD, **{key: None}), expected_cards = 2)
+
+
+def test_missing_facts_are_a_failure_rather_than_a_silence():
+ """A cycle that never ran the reading reports null, and null must not read
+ as "nothing wrong". This is the shape that let a payload be carried and
+ never executed for two rounds."""
+ payload = _payload()
+ assert payload.multi_gpu_failures(None, expected_cards = 2)
+ assert payload.multi_gpu_failures({}, expected_cards = 2)
+ broken = payload.multi_gpu_failures({"error": "ImportError: no unsloth"}, expected_cards = 2)
+ assert broken and "could not be read" in broken[0]
+
+
+def test_a_model_entirely_off_the_gpu_fails():
+ facts = dict(GOOD, parameters_by_device = {"cpu": 596049920}, cuda_devices_holding_parameters = [])
+ assert _payload().multi_gpu_failures(facts, expected_cards = 2)
+
+
+def test_the_spread_across_cards_is_RECORDED_and_not_required():
+ """The open question, and the rule must not pre-empt it.
+
+ Every two-card measurement this repo has is a LOAD and not a train
+ (unsloth-probe-vision-recon-c76ea3 saw a model split 897.7/1017.1 MB and
+ never called a trainer). So a single-card placement passes, a spread
+ passes, and the report carries which happened. When a session answers it,
+ tighten this deliberately rather than discovering the rule was already
+ asserting an answer nobody had.
+ """
+ payload = _payload()
+ one_card = dict(GOOD, cuda_devices_holding_parameters = ["cuda:0"])
+ spread = dict(
+ GOOD,
+ parameters_by_device = {"cuda:0": 300000000, "cuda:1": 296049920},
+ cuda_devices_holding_parameters = ["cuda:0", "cuda:1"],
+ )
+ assert payload.multi_gpu_failures(one_card, expected_cards = 2) == []
+ assert payload.multi_gpu_failures(spread, expected_cards = 2) == []
+
+
+def test_the_reading_is_taken_from_the_module_and_not_recomputed():
+ """`torch.cuda.device_count()` on both sides of the comparison is how a
+ rule ends up unable to fail: the binding is made once at import, so asking
+ torch again answers a different question and answers it agreeably."""
+ import ast
+
+ source = (SMOKE_DIR / "run_t4_smoke.py").read_text(encoding = "utf-8")
+ func = next(
+ n
+ for n in ast.walk(ast.parse(source))
+ if isinstance(n, ast.FunctionDef) and n.name == "multi_gpu_facts"
+ )
+ body = ast.unparse(func)
+ assert "from unsloth.kernels import utils" in body
+ assert "binding is torch.cuda.device" in body
+ assert "DEVICE_COUNT" in body
+
+
+# ------------------------------------------------------------ the leg itself
+
+
+def test_the_leg_asks_for_two_cards_and_the_BUILT_payload_enforces_it():
+ """Asserted through the generated notebook, not off the dataclass. A field
+ nothing emits reads like coverage and does nothing -- which is exactly how
+ a payload was carried and never run for two rounds."""
+ leg = legs.LEGS["multi_gpu"]
+ assert leg.all_cards is True
+ notebook = build_kernel.build_payload_notebook(
+ SMOKE_DIR, leg, unsloth_ref = "main", zoo_ref = "main"
+ )
+ source = "".join("".join(c["source"]) for c in notebook["cells"])
+ assert "device_count() == 2" in source, (
+ "the built payload does not require two visible GPUs, so a run the "
+ "driver pinned would measure the single-card branch and pass"
+ )
+ assert "--require-multi-gpu" in source
+ assert "--expected-cards" in source
+
+
+def test_every_OTHER_leg_still_requires_exactly_one():
+ """The pin check is what catches a driver that failed to pin, and widening
+ it for this leg must not widen it for the rest."""
+ for name, leg in legs.LEGS.items():
+ if name == "multi_gpu":
+ continue
+ notebook = build_kernel.build_payload_notebook(
+ SMOKE_DIR, leg, unsloth_ref = "main", zoo_ref = "main"
+ )
+ source = "".join("".join(c["source"]) for c in notebook["cells"])
+ assert "device_count() == 1" in source, name
+
+
+def test_the_leg_is_small_enough_to_co_tenant():
+ """The whole cost argument. At a whole card this leg would block gptoss
+ (12.78 of the 13.0 budget) for its entire life and the kernel would get
+ longer, which is the one thing this was built not to do."""
+ leg = legs.LEGS["multi_gpu"]
+ budget = 13.0 # CARD_VRAM_BUDGET_GB in build_kernel.py
+
+ # Expressed against the budget rather than a round number. The first
+ # version of this rule asserted `vram_gb <= 1.0`, which was a figure with no
+ # reasoning behind it, and it went red the moment the declaration was
+ # corrected from a guessed 0.7 to the measured 1.2 -- a guard failing on a
+ # measurement replacing a guess is the guard being wrong.
+ #
+ # What has to hold is that this leg can still SHARE with any ordinary leg,
+ # since running on both cards while excluding everything from both is the
+ # opposite of free.
+ others = [
+ other.vram_gb
+ for name, other in legs.LEGS.items()
+ if name != "multi_gpu" and other.vram_gb < budget / 2
+ ]
+ assert others
+ assert leg.vram_gb + max(others) <= budget, (
+ f"at {leg.vram_gb} GB on EVERY card this leg cannot share with a "
+ f"{max(others)} GB leg, so it takes both cards outright"
+ )
+ assert leg.vram_gb + legs.LEGS["gptoss"].vram_gb > budget, (
+ "gptoss must still be unable to share a card with this leg; if that "
+ "stopped being true the co-tenancy argument changed"
+ )
+
+
+def test_the_declaration_is_measured_and_not_copied_from_a_sibling():
+ """`vram_gb` was 0.7 for one commit, copied from the other Qwen legs. It is
+ wrong for this leg specifically -- it holds a CUDA context on BOTH cards --
+ and the repo-wide check could not say so, because it compares against
+ measured_vram.json and this leg was simply absent from it. A guard that
+ passes by finding nothing is the shape this directory keeps being caught
+ by."""
+ import json
+
+ measured = json.loads((SMOKE_DIR / "measured_vram.json").read_text(encoding = "utf-8"))[
+ "peak_reserved_gb"
+ ]
+ assert (
+ "multi_gpu" in measured
+ ), "the leg is not in the measured file, so the declaration check passes on it vacuously"
+ assert legs.LEGS["multi_gpu"].vram_gb >= measured["multi_gpu"]
+
+
+def test_it_does_not_export_a_gguf_and_the_reason_is_recorded():
+ """The bundle install_llama_cpp fetches for the notebook legs is the CPU
+ one -- on unsloth-probe-full-concurrent-417238 this model's llama-bench
+ reports `backend CPU` -- so a two-card tensor-split assertion here could
+ not fail. Adding the export anyway would cost ~40s of a thin margin and
+ buy a claim four other legs already make."""
+ leg = legs.LEGS["multi_gpu"]
+ assert "--export-gguf" not in leg.args
+ # The reason travels with the decision. A leg that simply lacks a flag
+ # invites someone to add it back in the hour they notice.
+ source = (ROOT / ".github" / "scripts" / "kaggle_t4_ci" / "legs.py").read_text(encoding = "utf-8")
+ entry = source.split('"multi_gpu": Leg(')[1].split('),\n "')[0]
+ assert "backend CPU" in entry or "CPU bundle" in entry
+
+
+# --------------------------------------------------------- driver scheduling
+
+
+def _driver_source() -> str:
+ payloads = {}
+ for name in ("default", "gptoss", "multi_gpu"):
+ leg = legs.LEGS[name]
+ payloads[f"t4_{leg.name}.ipynb"] = build_kernel.build_payload_notebook(
+ SMOKE_DIR, leg, unsloth_ref = "main", zoo_ref = "main"
+ )
+ driver = build_kernel.build_driver(
+ payloads,
+ per_run_timeout = 3600,
+ vram_source = {
+ f"t4_{legs.LEGS[n].name}.ipynb": legs.LEGS[n]
+ for n in ("default", "gptoss", "multi_gpu")
+ },
+ all_card = ("t4_Multi_GPU.ipynb",),
+ )
+ return "".join("".join(c["source"]) for c in driver["cells"])
+
+
+def test_the_all_card_payload_is_kept_out_of_the_card_queue():
+ source = _driver_source()
+ assert "ALL_CARD = (" in source
+ assert "t4_Multi_GPU.ipynb" in source.split("ALL_CARD = ")[1][:200]
+ assert "_queue = [(i, n) for i, n in enumerate(ORDER) if n not in ALL_CARD]" in source
+
+
+def test_it_runs_UNPINNED_or_it_measures_the_single_card_branch():
+ """`run_one(name, None, idx)`. An index here sets CUDA_VISIBLE_DEVICES and
+ the leg would take the nullcontext branch under a multi-GPU name."""
+ source = _driver_source()
+ lane = source.split("def _all_card_lane")[1].split("threads = []")[0]
+ assert "run_one(name, None, idx)" in lane
+ assert "run_one(name, 0" not in lane
+
+
+def test_the_reservation_is_all_or_nothing():
+ """A partial reservation holds a seat on card 0 while waiting for card 1 --
+ capacity nothing is using, and a deadlock against a big leg waiting for
+ card 0. The refusal path must not have mutated anything."""
+ source = _driver_source()
+ body = source.split("def _admit_all(name):")[1].split("def _release_all")[0]
+ check, commit = body.split("for g in range(N_GPU):")[1:3]
+ # Every rejection happens BEFORE any card is charged.
+ assert "return False" in check
+ assert "return False" not in commit
+ assert "card_load[g] += want" in commit
+ assert "card_load[g] += want" not in check
+
+
+def test_the_contention_is_global_and_the_vram_is_per_card():
+ """The two ledgers track different things, and charging the wrong one to a
+ card has now cost twice.
+
+ `card_load` is memory. An all-card leg really does hold a CUDA context on
+ each card -- 1.2 GB measured on unsloth-probe-multigpu-r2-a280e2 -- so it is
+ charged to every card.
+
+ `card_count` is a proxy for 4-vCPU CONTENTION; `MAX_LEGS_PER_CARD`'s own
+ comment says the legs "contend for CORES long before they contend for
+ memory". An unpinned process contends for those cores without occupying a
+ card, so it counts against the TOTAL bound and against no card in
+ particular.
+
+ Charged per card it was wrong in both available ways. On every card, the
+ driver simulation showed no card holding two legs at once. On ONE card, it
+ decided a placement it had nothing to do with and cost 188.7s on hardware:
+ in unsloth-probe-ab-with-multigpu-6169ca Studio was refused gpu0 (canary
+ plus this leg's count = the cap) and took gpu1, where the 1707s vision leg
+ had ~1080s left, so gpu1 carried both long payloads and gpu0 idled 622.6s.
+ The same kernel without the leg (unsloth-probe-ab-baseline-5leg-20db9c) ran
+ 1936.4s with 10.0s and 3.0s of idle.
+ """
+ source = _driver_source()
+ body = source.split("def _admit_all(name):")[1].split("def _release_all")[0]
+ assert "unpinned_count[0] += 1" in body
+ assert "card_count[" not in body.split("card_load[g] += want")[1], (
+ "the all-card leg is charging a card slot again, which is what put the "
+ "two longest payloads on one card and idled the other for 622.6s"
+ )
+ # The global bound has to be enforced on BOTH paths. In _admit_all only, the
+ # card queue fills up beside the unpinned leg and puts one more install than
+ # the bound allows onto four cores.
+ pinned = source.split("def _admit(gpu_index, name):")[1].split("def _release")[0]
+ for where, text in (("_admit", pinned), ("_admit_all", body)):
+ assert "unpinned_count[0] >= N_GPU * MAX_LEGS_PER_CARD" in text, where
+ release = source.split("def _release_all(name):")[1].split("\n\n")[0]
+ assert "unpinned_count[0] -= 1" in release
+
+
+def test_the_all_card_lane_is_joined_with_the_card_workers():
+ """It runs BESIDE the training legs, so anything concluding the cards are
+ free has to wait for it too."""
+ source = _driver_source()
+ assert "for t in all_card_threads:" in source
+ assert source.index("for t in all_card_threads:") < source.index(
+ "if AFTER_GPU and not AFTER_GPU_CONCURRENT:"
+ )
+
+
+def test_a_single_leg_dispatch_still_stands_down_on_one_card():
+ """`expected_gpus = min(len(payloads), SESSION_GPUS)` derives 1 for a
+ one-leg kernel, which is right for every leg but this one.
+
+ An all-card leg on a one-card allocation is an INFRASTRUCTURE fact, and
+ without this it arrives as a payload failure on `device_count() == 2` --
+ the exact confusion the shortfall guard's own comment says it exists to
+ prevent.
+ """
+ notebooks = build_kernel.build_kernel(
+ SMOKE_DIR,
+ ("multi_gpu",),
+ unsloth_ref = "main",
+ zoo_ref = "main",
+ extra_args = (),
+ per_run_timeout = 3600,
+ )
+ source = "".join("".join(c["source"]) for c in notebooks["cells"])
+ assert "EXPECTED_GPUS = 2" in source, (
+ "a one-leg multi_gpu dispatch accepts a single card, so the leg would "
+ "fail inside the payload instead of standing the kernel down"
+ )
+ # And an ordinary one-leg dispatch is unchanged.
+ other = build_kernel.build_kernel(
+ SMOKE_DIR,
+ ("default",),
+ unsloth_ref = "main",
+ zoo_ref = "main",
+ extra_args = (),
+ per_run_timeout = 3600,
+ )
+ assert "EXPECTED_GPUS = 1" in "".join("".join(c["source"]) for c in other["cells"])
+
+
+def test_the_weights_go_on_one_card_while_both_stay_visible():
+ """The two halves of the leg's configuration, and they pull opposite ways.
+
+ Visibility is what makes unsloth's DEVICE_COUNT > 1 bindings live, so the
+ leg must NOT be pinned. Placement is what makes training work, because a
+ sharded model does not train: on unsloth-probe-multigpu-r1-18beab
+ accelerate split Qwen3-0.6B across both T4s and step 0 died at
+ unsloth/models/llama.py:972 with `index is on cuda:0, different from other
+ tensors on cuda:1`.
+
+ Dropping `--single-device` puts a red in front of every PR for an upstream
+ fault; dropping `all_cards` silently turns the leg into a duplicate of
+ Default. Both are asserted here because either alone reads as configured.
+ """
+ leg = legs.LEGS["multi_gpu"]
+ assert leg.all_cards is True
+ assert "--single-device" in leg.args
+
+
+def test_single_device_reaches_the_child_and_sets_a_device_map():
+ import ast
+
+ source = (SMOKE_DIR / "run_t4_smoke.py").read_text(encoding = "utf-8")
+ tree = ast.parse(source)
+ train = next(
+ n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == "train_once"
+ )
+ body = ast.unparse(train)
+ assert "single_device" in body
+ assert "'device_map'] = {'': 0}" in body or '"device_map"] = {"": 0}' in body
+ # The CYCLE loads the model, so a parent that parsed this and kept it would
+ # shard anyway and the leg would die exactly as the probe did.
+ #
+ # The APPEND, not the string. `"--single-device" in unparse(main)` was the
+ # first version of this and it survived deleting the forwarding outright,
+ # because `ap.add_argument("--single-device", ...)` is in the same function
+ # -- an assertion satisfied by its own surrounding text.
+ main = next(n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == "main")
+ assert "cmd.append('--single-device')" in ast.unparse(main), (
+ "the flag is parsed but never forwarded, so the cycle child loads "
+ "sharded and dies at step 0 exactly as the probe did"
+ )
+
+
+def test_a_crash_mid_cycle_still_reports_what_was_measured():
+ """The probe measured everything this leg asserts, logged it in full, and
+ then LOST it: the cycle died in trainer.train(), wrote no report, and the
+ leg reported `multi_gpu: null` while the driver log held every number.
+
+ Reading the report alone said the measurement had not been taken, which is
+ the difference between "we did not look" and "we looked and it was fine".
+ """
+ import ast
+
+ source = (SMOKE_DIR / "run_t4_smoke.py").read_text(encoding = "utf-8")
+ tree = ast.parse(source)
+ facts = next(
+ n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == "multi_gpu_facts"
+ )
+ # Stashed as soon as there is anything to stash, not on the way out: a
+ # reading that lives only in a frame being unwound is a reading nobody has.
+ published = ast.unparse(facts)
+ assert "global _LAST_MULTI_GPU_FACTS" in published
+ assert "_LAST_MULTI_GPU_FACTS = facts" in published
+
+ main = next(n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == "main")
+ child = ast.unparse(main)
+ assert "except BaseException as exc" in child
+ assert "'multi_gpu': _LAST_MULTI_GPU_FACTS" in child
+ # A partial report must be impossible to mistake for a completed cycle, and
+ # the crash must still propagate -- swallowing it would turn a dead cycle
+ # into a green one.
+ assert "'partial': True" in child
+ assert "'cycle_error'" in child
+ assert "\n raise" in child
+
+
+def test_the_stash_is_populated_by_the_real_function_before_it_can_fail():
+ """Driven, not read. `multi_gpu_facts` walks the model, and a model that
+ raises mid-walk must still leave the device count behind."""
+ payload = _payload()
+ payload._LAST_MULTI_GPU_FACTS = None
+
+ class _Exploding:
+ def named_parameters(self):
+ raise RuntimeError("boom")
+
+ def modules(self):
+ return []
+
+ facts = payload.multi_gpu_facts(_Exploding())
+ assert payload._LAST_MULTI_GPU_FACTS is facts
+ assert "device_count" in payload._LAST_MULTI_GPU_FACTS
diff --git a/tests/kaggle/test_naive_trl_compare.py b/tests/kaggle/test_naive_trl_compare.py
new file mode 100644
index 00000000000..9c66c322d69
--- /dev/null
+++ b/tests/kaggle/test_naive_trl_compare.py
@@ -0,0 +1,169 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""The plain-TRL control arm, and the ways a "comparison" proves nothing.
+
+The rules under test are deliberately weak, and that is the point. Two library
+stacks do not produce one fp16 trajectory -- `frontier` measured transformers
+5.5.0 and 5.15.1 disagreeing at step 1 on identical weights, data and seed --
+so a guard that asserts the arms AGREE would be red on ordinary drift. These
+assert only what a comparison is entitled to: the control ran, it converged,
+and it ran the same number of steps as the arm it is printed beside.
+"""
+
+from __future__ import annotations
+
+import ast
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "t4_smoke"
+sys.path.insert(0, str(PAYLOAD))
+
+from naive_trl_compare import comparison_failures # noqa: E402
+
+
+def _trace(*losses):
+ return {"metrics": [{"step": i + 1, "loss": v} for i, v in enumerate(losses)]}
+
+
+def test_a_control_that_never_ran_is_a_failure_not_a_silence():
+ """The finding this file exists for. A missing arm must not read as a pass:
+ "no comparison" and "the comparison agreed" are opposite outcomes and only
+ one of them is evidence."""
+ assert comparison_failures(None, [{"loss": 1.0}])
+ assert comparison_failures({"error": "OOM"}, [{"loss": 1.0}])
+ assert "did not run" in comparison_failures({"error": "OOM"}, None)[0]
+
+
+def test_a_control_that_loaded_and_trained_nothing_is_a_failure():
+ broken = comparison_failures({"metrics": []}, [{"loss": 1.0}])
+ assert len(broken) == 1 and "reported no steps" in broken[0]
+
+
+def test_a_converging_control_passes():
+ assert comparison_failures(_trace(9.0, 5.0, 2.0), [{"loss": 1.0}] * 3) == []
+
+
+def test_a_flat_or_rising_control_fails():
+ assert comparison_failures(_trace(2.0, 5.0, 9.0), [{"loss": 1.0}] * 3)
+ assert comparison_failures(_trace(2.0, 2.0, 2.0), [{"loss": 1.0}] * 3)
+
+
+def test_a_non_finite_loss_fails_and_short_circuits():
+ broken = comparison_failures(_trace(9.0, float("nan"), 2.0), [{"loss": 1.0}] * 3)
+ assert len(broken) == 1 and "non-finite" in broken[0]
+
+
+def test_the_arms_must_have_run_the_same_number_of_steps():
+ """A control that quietly ran fewer steps is printed beside a full unsloth
+ trace as though the two were the same experiment."""
+ broken = comparison_failures(_trace(9.0, 5.0, 2.0), [{"loss": 1.0}] * 10)
+ assert broken and "different numbers of steps" in broken[0]
+
+
+def test_the_arms_are_never_asserted_equal():
+ """Mutation-proof against the obvious "improvement". Two wildly different
+ converging traces must PASS, because asserting agreement is what would make
+ this check red on every ordinary version bump."""
+ assert (
+ comparison_failures(
+ _trace(10.3222, 6.0, 1.0), [{"loss": 6.4367}, {"loss": 3.0}, {"loss": 0.5}]
+ )
+ == []
+ )
+
+
+def test_the_control_module_never_imports_unsloth():
+ """Asserted from the SOURCE, not from a convention. Anything that has
+ imported unsloth has had transformers, trl and peft patched underneath it
+ and is no longer a control; the comparison would be unsloth against itself
+ with extra steps, and it would look exactly like a real result."""
+ tree = ast.parse((PAYLOAD / "naive_trl_compare.py").read_text(encoding = "utf-8"))
+ imported = set()
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Import):
+ imported.update(a.name.split(".")[0] for a in node.names)
+ elif isinstance(node, ast.ImportFrom) and node.module:
+ imported.add(node.module.split(".")[0])
+ assert "unsloth" not in imported, sorted(imported)
+ assert "unsloth_zoo" not in imported, sorted(imported)
+
+
+def test_the_payload_runs_the_control_in_a_separate_process():
+ """A control imported into the parent would be patched by whatever the
+ parent imported. It must be spawned, and it must be spawned AFTER the
+ cycles: two 4bit models resident at once on a 14.56GB T4 is how a
+ comparison becomes an OOM blamed on the thing being compared."""
+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")
+ assert "naive_trl_compare.py" in src
+ assert "if args.compare_naive_trl:" in src
+ cycles_at = src.index("runs.append(json.loads(report_file.read_text")
+ spawn_at = src.index('"naive_trl_compare.py"')
+ assert cycles_at < spawn_at, "the control arm must be spawned after the cycles"
+
+
+def test_the_control_arm_loads_the_repo_unsloth_resolved():
+ """Not the name that was asked for, and the difference is an OOM.
+
+ `load_in_4bit=True` sends unsloth through FLOAT_TO_INT_MAPPER to a
+ pre-quantised `-unsloth-bnb-4bit` sibling. The plain path quantises the
+ ORIGINAL on the fly and has to materialise the 16bit checkpoint first. On
+ gemma-4-E2B-it that asked for 8.75 GiB on top of 7.25 GiB already resident
+ and died (kernel unsloth-probe-latestcompile-r3-cb1125).
+
+ Pointing both arms at the same weights is also the fairer comparison: the
+ question is what the two training stacks do, not which repo each loader
+ picks.
+ """
+ src = (ROOT / "tests" / "kaggle" / "t4_smoke" / "run_t4_smoke.py").read_text(encoding = "utf-8")
+ assert 'control_model = runs[0].get("resolved_checkpoint") or args.model' in src
+ assert '("--model", control_model),' in src
+
+
+def test_the_control_arm_uses_gradient_checkpointing():
+ """Leaving it off was unfair rather than neutral.
+
+ The unsloth arm runs with `gradient_checkpointing="unsloth"`, so a control
+ without it is measured with the single largest memory lever disabled on one
+ side only. On gemma-4-E2B-it that is the difference between a comparison and
+ an OOM: the control asked for 8.75 GiB on top of 8.96 GiB already resident,
+ on a 14.56 GiB card (kernel unsloth-probe-latestcompile-r4-e67ef2).
+ """
+ src = (PAYLOAD / "naive_trl_compare.py").read_text(encoding = "utf-8")
+ assert "use_gradient_checkpointing = True" in src
+ assert "gradient_checkpointing = True," in src
+ # Non-reentrant, or a PEFT model's inputs carry no grad and the backward
+ # fails with "element 0 of tensors does not require grad".
+ assert 'gradient_checkpointing_kwargs = {"use_reentrant": False}' in src
+
+
+def test_a_load_time_oom_can_be_reported_rather_than_failed():
+ """Measured: on gemma-4-E2B-it the plain arm asks for 8.75 GiB with 8.96 GiB
+ already resident on a 14.56 GiB T4, at LOAD -- `metrics` is absent, so no
+ step ever ran, and enabling gradient checkpointing changed the number not at
+ all. That is a statement about the card and the checkpoint, not about either
+ training stack."""
+ oom = {"error": "OutOfMemoryError: CUDA out of memory. Tried to allocate 8.75 GiB"}
+ assert comparison_failures(oom, [{"loss": 1.0}], allow_oom = True) == []
+
+
+def test_an_oom_is_still_a_failure_when_the_leg_did_not_opt_in():
+ oom = {"error": "OutOfMemoryError: CUDA out of memory"}
+ assert comparison_failures(oom, [{"loss": 1.0}])
+
+
+def test_an_oom_after_training_started_is_still_a_failure():
+ """The narrowness is the point. An OOM DURING training is a finding about
+ the run; only a failure to load is a fact about the card."""
+ oom = {
+ "error": "OutOfMemoryError: CUDA out of memory",
+ "metrics": [{"step": 1, "loss": 3.0}],
+ }
+ assert comparison_failures(oom, [{"loss": 1.0}], allow_oom = True)
+
+
+def test_a_non_oom_crash_is_never_excused():
+ crash = {"error": "ImportError: no module named trl"}
+ assert comparison_failures(crash, [{"loss": 1.0}], allow_oom = True)
diff --git a/tests/kaggle/test_prefetch_covers_the_wired_legs.py b/tests/kaggle/test_prefetch_covers_the_wired_legs.py
new file mode 100644
index 00000000000..5c5eebd0538
--- /dev/null
+++ b/tests/kaggle/test_prefetch_covers_the_wired_legs.py
@@ -0,0 +1,198 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""Every checkpoint a wired leg loads must be on the prefetch lane.
+
+The prefetch runs in the driver's own interpreter from t=0, on no card and no
+virtualenv, and it measured **~203 MB/s** on kernel
+unsloth-probe-prefetch-verify-9568-7a0bdd -- 12.5 GB of gpt-oss in 61.7s. A repo
+it does not name is downloaded by the leg instead, ON the card, while that card
+is allocated and idle.
+
+The failure is silent in the way this directory keeps being caught by: nothing
+is red, the leg simply takes longer, and a schedule built on the assumption that
+downloads are hidden is quietly wrong. It had already happened. `PREFETCH_REPOS`
+listed Qwen2.5-0.5B-Instruct and gpt-oss, while `vision_fla_compile` --
+the leg that SETS the makespan -- fetched its own 4.58 GB Qwen3.5-2B inline, and
+`default` fetched Qwen3-0.6B inline.
+
+Two details the rule has to respect, both learned the hard way:
+
+* a leg with no ``--model`` takes its payload's argparse default, so reading the
+ args alone reports the wrong checkpoint for every such leg;
+* ``load_in_4bit=True`` sends unsloth through FLOAT_TO_INT_MAPPER to a
+ ``-unsloth-bnb-4bit`` sibling, so warming the name in the args can warm a
+ cache the leg never reads. ``LOAD_REDIRECTS`` records the ones that are known
+ to differ.
+"""
+
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD_DIR = ROOT / "tests" / "kaggle" / "t4_smoke"
+sys.path.insert(0, str(ROOT / ".github" / "scripts"))
+
+from kaggle_t4_ci import legs # noqa: E402
+
+
+def _payload_default(entry: str) -> str | None:
+ """The `--model` default the payload itself carries."""
+ source = (PAYLOAD_DIR / entry).read_text(encoding = "utf-8")
+ if 'ap.add_argument("--model", default = DEFAULT_MODEL)' not in source:
+ return None
+ match = re.search(r'^DEFAULT_MODEL = "([^"]+)"', source, re.MULTILINE)
+ return match.group(1) if match else None
+
+
+def models_for(leg) -> set[str]:
+ args = list(leg.args)
+ if "--model" in args:
+ named = args[args.index("--model") + 1]
+ else:
+ named = _payload_default(leg.entry)
+ if not named:
+ return set()
+ # The declared name AND whatever it really resolves to on an sm_75 card.
+ return {named, legs.LOAD_REDIRECTS.get(named, named)}
+
+
+def test_every_wired_leg_loads_a_prefetched_checkpoint():
+ wired = [name for kernel in legs.KERNELS for name in kernel]
+ missing = {}
+ for name in wired:
+ wanted = models_for(legs.LEGS[name])
+ # The redirect target is what gets read, so satisfying either name is
+ # enough only when they are the same repo.
+ if wanted and not (wanted & set(legs.PREFETCH_REPOS)):
+ missing[name] = sorted(wanted)
+ assert not missing, (
+ "these wired legs download their checkpoint on an allocated card "
+ f"instead of on the free prefetch lane: {missing}. Add them to "
+ "PREFETCH_REPOS."
+ )
+
+
+def test_the_model_walk_reads_the_payload_default_and_not_only_the_args():
+ """Three of the five wired legs carry no --model at all. A rule that read
+ the args alone would report them as having no checkpoint and pass by
+ finding nothing, which is the shape of a guard that guards nothing."""
+ assert models_for(legs.LEGS["canary"]) == {"unsloth/Qwen2.5-0.5B-Instruct"}
+ assert "unsloth/gpt-oss-20b-unsloth-bnb-4bit" in models_for(legs.LEGS["gptoss"]), (
+ "the gpt-oss redirect is not being applied, so the prefetch would warm "
+ "the 16-bit repo that no sm_75 run ever loads"
+ )
+
+
+def test_the_critical_path_leg_is_fetched_before_the_one_with_slack():
+ """Order is not decoration: the lane fetches in the order given.
+
+ vision_fla_compile starts at t~21 and sets the makespan; gptoss is admitted
+ only when a card empties, around t~500 on the measured schedule. The
+ original order put gpt-oss first for margin when D was unknown; D is
+ measured now, so the leg with the least slack goes first.
+ """
+ order = list(legs.PREFETCH_REPOS)
+ assert order.index("unsloth/Qwen3.5-2B") < order.index(
+ "unsloth/gpt-oss-20b-unsloth-bnb-4bit"
+ ), "the leg that sets the makespan is queued behind the leg with 500s of slack"
+
+
+def test_nothing_is_prefetched_that_no_leg_reads():
+ """The other direction, and it costs bandwidth rather than time: a repo
+ nobody loads is a download the session pays for and never uses. Checked
+ against every leg, not only the wired ones, so a leg parked in UNWIRED can
+ keep its entry."""
+ loaded = set()
+ for leg in legs.LEGS.values():
+ loaded |= models_for(leg)
+ # Studio's own models are fetched by the Studio builder under its own
+ # HF_HOME, so they are deliberately not in this list.
+ stray = [repo for repo in legs.PREFETCH_REPOS if repo not in loaded]
+ assert not stray, f"prefetched but never loaded by any leg: {stray}"
+
+
+def test_every_redirect_target_is_prefetched_under_its_EXACT_name():
+ """The HF cache keys on the literal repo string.
+
+ `models--unsloth--qwen3-0.6b-unsloth-bnb-4bit` and
+ `models--unsloth--Qwen3-0.6B-unsloth-bnb-4bit` are different directories, so
+ prefetching the pretty spelling of a repo the loader asks for in lower case
+ warms a cache nobody reads and the session downloads it twice -- at full
+ cost, with no error and nothing red. Two hardware reports give the exact
+ strings; this asserts the lists agree with them character for character.
+ """
+ prefetch = set(legs.PREFETCH_REPOS)
+ wired = {name for kernel in legs.KERNELS for name in kernel}
+ for leg_name in wired:
+ leg = legs.LEGS[leg_name]
+ args = list(leg.args)
+ named = (
+ args[args.index("--model") + 1] if "--model" in args else _payload_default(leg.entry)
+ )
+ if named is None or named not in legs.LOAD_REDIRECTS:
+ continue
+ target = legs.LOAD_REDIRECTS[named]
+ assert target in prefetch, (
+ f"{leg_name} loads {target!r}, which is not prefetched under that "
+ f"exact string; a case-different entry does not warm the same cache"
+ )
+
+
+def test_the_lower_case_qwen3_redirect_is_not_tidied_away():
+ """It looks like a typo and it is a measurement. Written down so the next
+ reader corrects the capitals in the report, not in this file."""
+ assert legs.LOAD_REDIRECTS["unsloth/Qwen3-0.6B"] == "unsloth/qwen3-0.6b-unsloth-bnb-4bit"
+
+
+def test_a_blob_is_counted_once_not_once_per_symlink(tmp_path, monkeypatch):
+ """`snapshot_download` writes each file once under `blobs/` and links to it
+ from `snapshots/`, so an `os.stat` walk follows the link and counts the same
+ bytes twice.
+
+ Measured on kernel unsloth-probe-prefetch-verify-9568-7a0bdd: gpt-oss came
+ back as 25109731082 bytes for a ~12.5 GB checkpoint, and the reported
+ 407.0 MB/s was really ~203. This is the number the second-wave ordering and
+ the makespan argument rest on, so a 2x is not cosmetic.
+
+ Drives the REAL generated cell body rather than a copy of the walk: the
+ function under test only ever exists inside that f-string, and a
+ reimplementation here would pass while the shipped one doubles.
+ """
+ import importlib.util
+ import os
+
+ spec = importlib.util.spec_from_file_location(
+ "kaggle_prefetch_under_test", ROOT / ".github" / "scripts" / "kaggle_prefetch.py"
+ )
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+
+ folder = tmp_path / "hub" / "models--org--model"
+ blobs = folder / "blobs"
+ snapshot = folder / "snapshots" / "rev"
+ blobs.mkdir(parents = True)
+ snapshot.mkdir(parents = True)
+ for index in range(3):
+ blob = blobs / f"sha{index}"
+ blob.write_bytes(b"x" * 1000)
+ (snapshot / f"shard{index}.safetensors").symlink_to(blob)
+ # A config is a real file in both places on a filesystem without symlinks,
+ # which is the other way the same bytes get counted twice.
+ (blobs / "cfg").write_bytes(b"y" * 10)
+ os.link(blobs / "cfg", snapshot / "config.json")
+
+ source = module.prefetch_cell(repos = [("org/model", None)], hf_home = str(tmp_path))
+ match = re.search(r"def _repo_bytes\(repo\):.*?\n\ndef ", source, re.S)
+ assert match, "the generated cell no longer defines _repo_bytes"
+ namespace = {"os": os}
+ exec(match.group(0)[: -len("\n\ndef ")], namespace) # noqa: S102
+
+ monkeypatch.setenv("HF_HOME", str(tmp_path))
+ assert namespace["_repo_bytes"]("org/model") == 3010, (
+ "the walk counts a blob once per link to it, so every reported size and "
+ "MB/s in the prefetch evidence is inflated"
+ )
diff --git a/tests/kaggle/test_studio_api_key.py b/tests/kaggle/test_studio_api_key.py
new file mode 100644
index 00000000000..a428c442176
--- /dev/null
+++ b/tests/kaggle/test_studio_api_key.py
@@ -0,0 +1,109 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""Minting a Studio API key, and the two ways that check proves nothing.
+
+Creating a key proves the endpoint returns a string. Whether that string
+authenticates anything is a separate question, and it is the one that matters:
+a key Studio issues and then rejects fails in a user's integration rather than
+here.
+
+**Vacuity 1: passing on the session token.** If the bearer is not swapped, the
+request succeeds because the session is still logged in and the key is never
+exercised.
+
+**Vacuity 2: a server that ignores the header.** Then any key "works", including
+one that was never issued. The corrupted-key call is what rules that out, and
+without it the positive check is satisfied by a server with no auth at all.
+"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"
+SRC = PAYLOAD.read_text(encoding = "utf-8")
+
+
+def _assert_api_key_body() -> str:
+ tree = ast.parse(SRC)
+ func = next(
+ node
+ for node in ast.walk(tree)
+ if isinstance(node, ast.FunctionDef) and node.name == "assert_api_key"
+ )
+ return ast.get_source_segment(SRC, func) or ""
+
+
+def test_the_key_is_created_through_the_real_endpoint():
+ body = _assert_api_key_body()
+ assert '"/api/auth/api-keys"' in body
+
+
+def test_the_session_token_is_set_aside_while_the_key_is_driven():
+ """Vacuity 1. Leaving the session token in place makes the call succeed for
+ a reason that has nothing to do with the key."""
+ body = _assert_api_key_body()
+ assert "saved = self.studio.token" in body
+ assert "self.studio.token = saved" in body, "the session token must be restored"
+
+ # Structural, not a substring. `self.studio.token = raw_key[:-4] + "0000"`
+ # CONTAINS the string "self.studio.token = raw_key", so a substring check
+ # passes even when the swap line is deleted and only the corrupted-key
+ # assignment remains -- which is exactly the mutation this guard exists to
+ # catch, and it survived the first version of this test.
+ tree = ast.parse(body)
+ swaps = [
+ node
+ for node in ast.walk(tree)
+ if isinstance(node, ast.Assign)
+ and isinstance(node.value, ast.Name)
+ and node.value.id == "raw_key"
+ and any(isinstance(t, ast.Attribute) and t.attr == "token" for t in node.targets)
+ ]
+ assert swaps, (
+ "the bearer is never set to the raw key itself, so the positive check "
+ "runs on whatever token was already there"
+ )
+
+
+def test_a_corrupted_key_must_be_rejected():
+ """Vacuity 2. Without this, a server that ignores the Authorization header
+ entirely passes the positive check."""
+ body = _assert_api_key_body()
+ assert "bad_key_status" in body
+ assert "a corrupted API key was accepted" in body
+
+
+def test_the_raw_key_is_registered_as_a_secret_before_it_is_used():
+ """It reaches request headers and, on failure, error strings that travel
+ home in the evidence bundle."""
+ body = _assert_api_key_body()
+ add_at = body.index("self.secrets.add(raw_key)")
+ use_at = body.index("self.studio.token = raw_key")
+ assert add_at < use_at, "the key is used before it is registered for scrubbing"
+
+
+def test_the_session_token_is_restored_even_when_the_call_raises():
+ """A finally, not a trailing statement: an exception mid-check would
+ otherwise leave the session authenticated as a corrupted API key and every
+ later assertion would fail for the wrong reason."""
+ tree = ast.parse(_assert_api_key_body())
+ tries = [n for n in ast.walk(tree) if isinstance(n, ast.Try) and n.finalbody]
+ restores = [
+ t
+ for t in tries
+ if "self.studio.token = saved" in "".join(ast.dump(s) for s in t.finalbody)
+ or any("saved" in ast.dump(s) for s in t.finalbody)
+ ]
+ assert restores, "the token restore is not in a finally"
+
+
+def test_it_runs_before_the_long_gpu_work():
+ """It needs only a logged-in session. After a 20-minute training run, a
+ training failure would hide whether API keys work at all."""
+ api_at = SRC.index("self.assert_api_key()")
+ train_at = SRC.index("trained = self.assert_training()")
+ assert api_at < train_at
diff --git a/tests/kaggle/test_studio_cli_run.py b/tests/kaggle/test_studio_cli_run.py
new file mode 100644
index 00000000000..a95c6ca6c1b
--- /dev/null
+++ b/tests/kaggle/test_studio_cli_run.py
@@ -0,0 +1,221 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""`unsloth run`: the headless model server, and what a banner does not prove.
+
+`unsloth run` is a different launch from `unsloth studio`, and nothing in CI
+covered it. It starts the backend, waits for health, mints an API key
+IN-PROCESS, and only then loads the model over HTTP. Any of those four steps
+can fail while the command still prints a banner, so the rules here are about
+what came back rather than what was printed.
+
+The two that carry the most weight, because each closes a hole the other
+leaves:
+
+**GPU residency across the launch.** A GGUF server that fell back to the CPU
+serves text perfectly well, so "it answered" is not evidence it reached the
+card. The measurement is PER-PROCESS rather than a device total: under
+--studio-concurrent a training leg shares the card, and on kernel
+unsloth-probe-full-concurrent-417238 the device delta read -182.0 MiB while
+the same report showed this launch holding 2628 MiB. A shared counter cannot
+attribute. `cli_run_gpu_failure` is a pure function precisely so the rules
+below can DRIVE it with those numbers rather than describe it.
+
+**A corrupted key must be REFUSED.** Without it, a server that ignores the
+Authorization header entirely satisfies "the minted key authenticated".
+"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"
+SRC = PAYLOAD.read_text(encoding = "utf-8")
+
+
+def _func(name: str) -> ast.FunctionDef:
+ for cls in ast.walk(ast.parse(SRC)):
+ if not isinstance(cls, ast.ClassDef):
+ continue
+ for node in cls.body:
+ if isinstance(node, ast.FunctionDef) and node.name == name:
+ return node
+ raise AssertionError(f"no method named {name!r}")
+
+
+def _body(name: str = "assert_cli_run") -> str:
+ return ast.get_source_segment(SRC, _func(name)) or ""
+
+
+def test_the_assertion_exists_and_is_driven_from_the_run():
+ assert _body()
+ assert "self.assert_cli_run()" in _body("execute")
+
+
+def test_it_runs_after_the_ui_phase_has_stopped_the_server():
+ """Not a preference. `unsloth run` starts a SECOND backend against the same
+ studio home, and two backends sharing one home's state is a configuration
+ nobody ships. It also makes the VRAM delta meaningless: a card still
+ holding the first server's model cannot show this launch's growth."""
+ body = _body("execute")
+ ui_at = body.index("self.assert_chat_ui()")
+ cli_at = body.index("self.assert_cli_run()")
+ assert ui_at < cli_at, "the CLI launch must come after the UI phase stops the server"
+
+
+def test_it_uses_its_own_port():
+ """The first server's port may still be in TIME_WAIT, and a bind failure
+ there would read as a broken CLI."""
+ assert "port = self.args.port + 1" in _body()
+
+
+def test_no_public_url_is_ever_opened_from_ci():
+ """--secure implies a Cloudflare quick tunnel, which publishes this server
+ to the internet from a CI kernel. --no-cloudflare is explicit rather than
+ relying on the default, because a default is a thing that changes."""
+ body = "".join(_body().split()) # formatter-proof; see the cloudflare guards
+ assert '"--no-cloudflare",' in body
+ assert '"--secure"' not in body
+
+
+def test_the_key_comes_from_the_marker_the_cli_itself_prints():
+ body = "".join(_body().split())
+ assert '"--start-api-key-marker",' in body
+ assert '"UNSLOTH_START_API_KEY:"intext' in body # whitespace-stripped
+
+
+def test_the_key_is_registered_as_a_secret_before_anything_reads_the_log():
+ """The log is packed into the evidence bundle. `redacted()` is what keeps
+ the key out of it, and it can only redact a secret it has been told about,
+ so the registration has to happen at the moment the key is parsed."""
+ src = _body()
+ parse_at = src.index('text.split("UNSLOTH_START_API_KEY:"')
+ add_at = src.index("self.secrets.add(api_key)")
+ assert parse_at < add_at, "the key must be registered where it is parsed"
+ # And the log must actually be in the bundle, or nothing redacts it.
+ assert '"unsloth_run.log",' in _body("emit_evidence")
+
+
+def test_gpu_use_is_measured_and_an_unmeasurable_reading_is_a_failure():
+ """ "nvidia-smi did not answer" and "the model was on the GPU" are opposite
+ outcomes; treating the first as a pass is the exact shape this directory
+ has been caught by before."""
+ body = _body()
+ assert "baseline = nvidia_used_mib()" in body
+ assert "settled = nvidia_used_mib()" in body
+ # The verdict itself moved into `cli_run_gpu_failure` when the device delta
+ # stopped being a valid ruler under --studio-concurrent, so it is DRIVEN
+ # rather than grepped -- see the rules at the end of this file. What stays
+ # here is that the assertion consults it and reports what it says.
+ assert "cli_run_gpu_failure(" in body
+ assert "failures.append(failure)" in body
+ verdict = _verdict()
+ assert verdict(None, None, None, None)[0], "an unmeasurable reading passed"
+
+
+def test_a_corrupted_key_must_be_refused():
+ """Without this, a server ignoring the Authorization header entirely
+ satisfies the claim that the minted key authenticated."""
+ func = _func("assert_cli_run")
+ guarded = [
+ n
+ for n in ast.walk(func)
+ if isinstance(n, ast.If)
+ and "bad_key_status" not in ast.unparse(n.test)
+ and "code < 400" in ast.unparse(n.test)
+ ]
+ assert guarded, "nothing refuses a corrupted key"
+
+
+def test_the_child_is_always_torn_down():
+ """A `unsloth run` left alive holds a card and a port for the rest of the
+ session, and the kernel's next phase reads that as its own failure."""
+ func = _func("assert_cli_run")
+ tries = [n for n in ast.walk(func) if isinstance(n, ast.Try) and n.finalbody]
+ assert tries, "the teardown must be in a finally, or a raised assertion leaks the server"
+ finals = "\n".join(ast.unparse(n) for t in tries for n in t.finalbody)
+ assert "proc.terminate()" in finals
+ assert "proc.kill()" in finals, "terminate alone leaves a hung server running"
+
+
+def test_the_vram_sample_comes_AFTER_a_served_completion():
+ """`unsloth run` prints its API key while it is still starting.
+
+ Sampling there read 0.0 MiB of growth on kernel
+ unsloth-probe-studio-full2-815a0c, on a launch whose own log says
+ `Starting llama-server: ... -ngl -1 --fit off` -- Studio asking for every
+ layer on the card. A completion that came back is the cheap proof the
+ weights are resident, so the ruler has to go after it or the check measures
+ a race.
+ """
+ func = _func("assert_cli_run")
+ src = ast.get_source_segment(SRC, func) or ""
+ sample_at = src.index('detail["vram_after_mib"]')
+ completion_at = src.index('detail["completion_status"]')
+ assert completion_at < sample_at, (
+ "VRAM is sampled before a completion has been served, so a slow load "
+ "reads as a CPU fallback"
+ )
+
+
+def _verdict():
+ """The real function, loaded by path rather than reimplemented."""
+ import importlib.util
+
+ spec = importlib.util.spec_from_file_location("_studio_payload_cli", PAYLOAD)
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module.cli_run_gpu_failure
+
+
+def test_a_co_tenant_freeing_memory_does_not_read_as_a_CPU_fallback():
+ """The exact numbers from unsloth-probe-full-concurrent-417238.
+
+ Device VRAM 2816 -> 2634, a delta of -182.0, while `nvidia-smi` shows this
+ launch's own pid holding 2628 MiB. Under the old device-delta rule that was
+ a failure saying `unsloth run` served from the CPU; the model was on the
+ card the whole time and a training leg on the same card freed memory inside
+ the window. This is the regression guard for that reading.
+ """
+ failure, detail = _verdict()({}, {6841: 2628}, 2816.0, 2634.0)
+ assert failure is None, failure
+ assert detail["process_vram_mib"] == 2628
+ # The device delta is still RECORDED -- it is evidence, it is just not the
+ # verdict -- and it is still the number that misled.
+ assert detail["vram_delta_mib"] == -182.0
+
+
+def test_a_co_tenant_ALREADY_on_the_card_cannot_satisfy_the_claim():
+ """Counting every process would pass on a card a training leg is using and
+ a server that never left the CPU. Only pids that APPEARED count."""
+ failure, detail = _verdict()({99: 12000}, {99: 12000}, 100.0, 100.0)
+ assert failure and "served from the CPU" in failure
+ assert detail["process_vram_mib"] == 0
+
+
+def test_a_real_cpu_fallback_still_fails():
+ """The case the assertion exists for: the launch answered, and no process
+ of its own ever appeared on the GPU."""
+ failure, _ = _verdict()({99: 12000}, {99: 12000, 4242: 3}, 100.0, 101.0)
+ assert failure and "served from the CPU" in failure
+
+
+def test_the_device_delta_is_the_fallback_only_when_processes_are_unreadable():
+ """An nvidia-smi that answers a total but cannot enumerate apps still gets a
+ verdict rather than a silent pass."""
+ verdict = _verdict()
+ assert verdict(None, None, 100.0, 4000.0)[0] is None
+ failure, _ = verdict(None, None, 100.0, 110.0)
+ assert failure and "could not enumerate processes" in failure
+ assert verdict(None, None, None, None)[0] == (
+ "nvidia-smi did not answer, so GPU use is unmeasured"
+ )
+
+
+def test_the_before_sample_is_taken_before_the_launch():
+ """An `apps_before` read after the server started would contain the server,
+ so nothing would ever have `appeared` and every run would fail."""
+ body = _body()
+ assert body.index("apps_before = nvidia_compute_apps()") < body.index("subprocess.Popen")
diff --git a/tests/kaggle/test_studio_cloudflare.py b/tests/kaggle/test_studio_cloudflare.py
new file mode 100644
index 00000000000..4aaeb40d3fb
--- /dev/null
+++ b/tests/kaggle/test_studio_cloudflare.py
@@ -0,0 +1,163 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""The quick tunnel: the one assertion here that reaches the public internet.
+
+Because it does, what it claims is narrow and what it refuses to claim is
+written down.
+
+Three rules, and the third is the one that makes opening a public URL from a CI
+machine defensible at all: the tunnel must REFUSE an unauthenticated request.
+Without it this check would prove that anyone holding the URL can drive
+inference on the box, and call that a pass.
+
+Two details are load-bearing rather than stylistic:
+
+* `--host 0.0.0.0`. Studio raises a quick tunnel for WILDCARD binds only; on
+ 127.0.0.1 there is nothing to publish, no URL is printed, and a working
+ feature reads as a broken one.
+* the negative lookahead on `api.trycloudflare.com`. That host appears in
+ cloudflared's own FAILURE lines ("failed to request quick Tunnel: Post
+ https://api.trycloudflare.com/tunnel"), so a naive match extracts a URL from
+ the message that says there is no URL. Studio's own matcher carries the same
+ lookahead, and this one is checked against both strings.
+"""
+
+from __future__ import annotations
+
+import ast
+import re
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"
+SRC = PAYLOAD.read_text(encoding = "utf-8")
+
+
+def _func(name: str) -> ast.FunctionDef:
+ for cls in ast.walk(ast.parse(SRC)):
+ if not isinstance(cls, ast.ClassDef):
+ continue
+ for node in cls.body:
+ if isinstance(node, ast.FunctionDef) and node.name == name:
+ return node
+ raise AssertionError(f"no method named {name!r}")
+
+
+def _body(name: str = "assert_cloudflare") -> str:
+ return ast.get_source_segment(SRC, _func(name)) or ""
+
+
+def test_the_assertion_exists_and_is_driven_from_the_run():
+ assert _body()
+ assert "self.assert_cloudflare()" in _body("execute")
+
+
+def _flat(text: str) -> str:
+ """Whitespace-stripped, because the repo's formatter reflows an argument
+ list to one entry per line and a guard matching the unformatted spelling
+ goes red on a reformat rather than on a regression. That has happened three
+ times in this payload now."""
+ return "".join(text.split())
+
+
+def test_it_binds_a_wildcard_or_there_is_nothing_to_publish():
+ body = _flat(_body())
+ assert '"--host","0.0.0.0",' in body
+ assert '"--cloudflare",' in body
+
+
+def test_the_url_matcher_rejects_cloudflareds_own_failure_host():
+ """Executed, not eyeballed. The failure line contains a trycloudflare URL
+ and means the opposite of a tunnel."""
+ body = _body()
+ literal = re.search(r"re\.search\((r\"[^\"]+\")", body)
+ assert literal, "no URL matcher found"
+ pattern = re.compile(ast.literal_eval(literal.group(1)))
+ assert pattern.search("Your quick Tunnel: https://brave-cat-runs.trycloudflare.com")
+ assert not pattern.search(
+ 'failed to request quick Tunnel: Post "https://api.trycloudflare.com/tunnel"'
+ ), "the failure host was accepted as a working tunnel URL"
+
+
+def test_the_tunnel_must_actually_serve():
+ """ "a URL was printed" and "a URL that works" are different claims. A
+ tunnel whose edge never registers answers 530."""
+ func = _func("assert_cloudflare")
+ assert any(
+ isinstance(n, ast.If) and ast.unparse(n.test) == "code != 200" for n in ast.walk(func)
+ ), "nothing checks that the public URL answers"
+
+
+def test_an_unauthenticated_request_through_the_tunnel_must_be_refused():
+ """The rule that makes opening the URL defensible. Asserted structurally so
+ a message rewrite cannot satisfy it."""
+ func = _func("assert_cloudflare")
+ unauth = [
+ n
+ for n in ast.walk(func)
+ if isinstance(n, ast.Call)
+ and any(k.arg == "auth" and k.value.value is False for k in n.keywords)
+ ]
+ assert len(unauth) >= 2, (
+ "both the health probe and the inference probe must go through the "
+ "tunnel WITHOUT credentials"
+ )
+ assert any(
+ isinstance(n, ast.If) and "code < 400" in ast.unparse(n.test) for n in ast.walk(func)
+ ), "an accepted unauthenticated request must be a failure"
+
+
+def test_the_no_tunnel_excuse_is_narrow():
+ """Reported-not-failed applies ONLY when nothing was published, because
+ then there is nothing to have gone wrong with. Once a URL exists, every
+ rule above is live."""
+ func = _func("assert_cloudflare")
+ excuses = [
+ n for n in ast.walk(func) if isinstance(n, ast.If) and ast.unparse(n.test) == "url is None"
+ ]
+ assert excuses, "the excuse branch is gone or is no longer gated on the URL"
+ for node in excuses:
+ # node.body only. Walking the whole If would sweep in the `else`
+ # branch, which is where every real failure lives, and the guard would
+ # then fail on correct code -- it did, the first time this was written.
+ appended = [
+ n
+ for stmt in node.body
+ for n in ast.walk(stmt)
+ if isinstance(n, ast.Call)
+ and isinstance(n.func, ast.Attribute)
+ and n.func.attr == "append"
+ and isinstance(n.func.value, ast.Name)
+ and n.func.value.id == "failures"
+ ]
+ assert not appended, "the no-tunnel branch reports; it does not fail"
+ body = _body()
+ assert "no_tunnel_reason" in body, "a reason must be carried from the log"
+
+
+def test_the_public_url_is_registered_as_a_secret():
+ """It is a live public route to this machine and the artifact is read by
+ people who are not running it."""
+ body = _body()
+ assert "self.secrets.add(url)" in body
+ assert '"unsloth_cloudflare.log",' in _body(
+ "emit_evidence"
+ ), "a log nobody packs is a log nobody redacts"
+
+
+def test_the_child_is_always_torn_down():
+ """A tunnel left up outlives the assertion and keeps the machine public."""
+ func = _func("assert_cloudflare")
+ finals = "\n".join(
+ ast.unparse(n) for t in ast.walk(func) if isinstance(t, ast.Try) for n in t.finalbody
+ )
+ assert "proc.terminate()" in finals
+ assert "proc.kill()" in finals
+
+
+def test_it_can_be_switched_off():
+ """The only assertion here that reaches the public internet should be
+ refusable without editing the payload."""
+ assert '"--no-cloudflare-check"' in SRC
+ assert "self.args.cloudflare_check" in _body("execute")
diff --git a/tests/kaggle/test_studio_code_execution.py b/tests/kaggle/test_studio_code_execution.py
new file mode 100644
index 00000000000..4df1b86bea7
--- /dev/null
+++ b/tests/kaggle/test_studio_code_execution.py
@@ -0,0 +1,165 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""Studio's local `python` tool must be EXECUTED, not merely offered.
+
+The vacuity this file exists against is a specific one. `assert_tool_calling`
+already proves the model can emit a tool call, and a weather tool is never run
+by Studio at all -- the caller runs it. The local `python` tool is different:
+Studio executes it in a per-session sandbox, and the failure worth catching is
+a loop that hands the model the schema, never runs the call, and lets the model
+narrate a plausible result. From the reply text that is indistinguishable from
+success, because a language model will happily report the output of code that
+never ran.
+
+So the pass condition must come off the FILESYSTEM. A per-run token is written
+by the executed code into `<studio home>/sandbox`, and only real execution puts
+those bytes on this disk.
+
+One configuration detail is load-bearing rather than incidental:
+`routes/inference.py` rejects a local python/terminal tool with a 400 under
+`permission_mode` `ask`, and under `auto` or the omitted default, because there
+is no confirmation channel on this path. A payload that left it unset would
+fail on configuration and read as a broken tool.
+"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"
+SRC = PAYLOAD.read_text(encoding = "utf-8")
+
+
+def _func(name: str) -> ast.FunctionDef:
+ """The METHOD of that name, not a module-level function.
+
+ `run` is both: a module-level `subprocess` helper and the harness's own
+ sequencer. Walking the whole tree finds the helper first, and every
+ assertion about the sequence then reads a body that never mentions it.
+ """
+ tree = ast.parse(SRC)
+ for cls in ast.walk(tree):
+ if not isinstance(cls, ast.ClassDef):
+ continue
+ for node in cls.body:
+ if isinstance(node, ast.FunctionDef) and node.name == name:
+ return node
+ raise AssertionError(f"no method named {name!r}")
+
+
+def _body(name: str = "assert_code_execution") -> str:
+ return ast.get_source_segment(SRC, _func(name)) or ""
+
+
+def test_the_assertion_exists_and_is_driven_from_the_run():
+ body = _body()
+ assert body
+ run = _body("execute")
+ assert "self.assert_code_execution()" in run, "an assertion nothing calls is not coverage"
+
+
+def test_the_python_tool_is_the_one_requested():
+ """`enabled_tools` omitted means ALL local tools, which would let the model
+ satisfy this with `terminal` and say nothing about `python`."""
+ body = _body()
+ assert "enable_tools = True" in body
+ assert 'enabled_tools = ["python"]' in body
+
+
+def test_the_permission_mode_is_one_the_local_python_tool_survives():
+ """Not a style choice. `ask` is rejected outright for a local python tool,
+ and so are `auto` and the omitted default, with a 400 -- there is no
+ confirmation channel on this path. Either of those would fail the run on
+ configuration while looking like a broken tool."""
+ body = _body()
+ assert 'permission_mode = "off"' in body
+ assert 'permission_mode = "ask"' not in body
+ assert 'permission_mode = "auto"' not in body
+
+
+def test_the_token_is_minted_per_run_and_not_hardcoded():
+ """A fixed token would be satisfied by a file an EARLIER run left in the
+ sandbox, which is a green tick for code that did not run this time."""
+ body = _body()
+ assert "secrets_module.token_hex" in body
+ call = next(
+ node
+ for node in ast.walk(_func("assert_code_execution"))
+ if isinstance(node, ast.Call)
+ and isinstance(node.func, ast.Attribute)
+ and node.func.attr == "token_hex"
+ )
+ assert call is not None
+
+
+def test_the_verdict_is_read_off_the_filesystem_and_not_off_the_reply():
+ """The rule this whole file is about. The failure must be raised by the
+ absence of a written file; a check on the reply text would pass on a model
+ that narrated an execution that never happened.
+
+ Asserted structurally rather than by message matching, because a message
+ is satisfied by its own surrounding text.
+ """
+ func = _func("assert_code_execution")
+
+ # Every `failures.append` that decides the verdict, and what it is guarded by.
+ guarded_by_written = False
+ for node in ast.walk(func):
+ if not isinstance(node, ast.If):
+ continue
+ test = ast.unparse(node.test)
+ appends = [
+ n
+ for n in ast.walk(node)
+ if isinstance(n, ast.Call)
+ and isinstance(n.func, ast.Attribute)
+ and n.func.attr == "append"
+ ]
+ if not appends:
+ continue
+ if "written" in test:
+ guarded_by_written = True
+ assert isinstance(node.test, ast.UnaryOp) and isinstance(
+ node.test.op, ast.Not
+ ), "the failure must fire when NO file carries the token"
+ assert guarded_by_written, (
+ "nothing in this assertion depends on a file being written, so it "
+ "would pass on a reply that merely claimed the code ran"
+ )
+
+ # And the reply is recorded but never judged.
+ for node in ast.walk(func):
+ if isinstance(node, ast.If) and "reply" in ast.unparse(node.test):
+ raise AssertionError(
+ "the verdict branches on the model's prose, which is exactly "
+ "what an unexecuted tool call can fake"
+ )
+
+
+def test_the_search_reads_content_rather_than_only_a_filename():
+ """A small local model rewording the path is not a Studio defect. What
+ cannot be faked is the token's BYTES landing on disk, so the content is
+ what the search is entitled to rely on."""
+ body = _body()
+ assert "read_text" in body
+ assert "token in body" in body
+
+
+def test_it_looks_under_the_studio_home_sandbox_and_not_a_stray_root():
+ """`sandbox_root()` puts the per-session working directories under the
+ studio home precisely so `UNSLOTH_STUDIO_HOME` keeps them together. Reading
+ a fixed `~/studio_sandbox` would search a directory this run never wrote
+ to, and report a correct execution as a failure."""
+ body = _body()
+ assert 'self.studio_home / "sandbox"' in body
+
+
+def test_the_cpu_fallback_records_the_assertion_rather_than_omitting_it():
+ """A skipped assertion that leaves no record is indistinguishable from one
+ that never existed, and the summary would be short by one line rather than
+ red."""
+ run = _body("execute")
+ assert '"code_execution",' in run, "the non-GPU branch must record code_execution explicitly"
diff --git a/tests/kaggle/test_studio_compaction.py b/tests/kaggle/test_studio_compaction.py
new file mode 100644
index 00000000000..4429f55f22f
--- /dev/null
+++ b/tests/kaggle/test_studio_compaction.py
@@ -0,0 +1,244 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""Context compaction, and why one half of the check is worthless alone.
+
+Studio reports the fit on the completion itself, as `context_truncated` with
+`dropped_messages`, so a conversation past the window can be shown to have been
+SHORTENED rather than refused, without a browser.
+
+The rule is a pair, deliberately:
+
+* an over-length conversation returns 200 with `dropped_messages > 0`;
+* a two-message conversation drops nothing.
+
+The first alone passes on a server that reports truncation unconditionally,
+which is indistinguishable from working. The second alone passes on a server
+that never compacts and returns a context-length error instead. Only together
+do they say the field tracks length.
+
+It also has to run where the window is KNOWN. `assert_server_flags` reloads the
+model pinned to `--studio-ctx`; a compaction check before that is aimed at a
+context length nobody set.
+"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"
+SRC = PAYLOAD.read_text(encoding = "utf-8")
+
+
+def _func(name: str) -> ast.FunctionDef:
+ for cls in ast.walk(ast.parse(SRC)):
+ if not isinstance(cls, ast.ClassDef):
+ continue
+ for node in cls.body:
+ if isinstance(node, ast.FunctionDef) and node.name == name:
+ return node
+ raise AssertionError(f"no method named {name!r}")
+
+
+def _body(name: str = "assert_compaction") -> str:
+ return ast.get_source_segment(SRC, _func(name)) or ""
+
+
+def test_the_assertion_exists_and_is_driven_from_the_run():
+ assert _body()
+ assert "self.assert_compaction()" in _body("execute")
+
+
+def test_it_runs_after_the_window_is_pinned():
+ body = _body("execute")
+ flags_at = body.index("self.assert_server_flags()")
+ ours_at = body.index("self.assert_compaction()")
+ assert flags_at < ours_at, (
+ "the reload in assert_server_flags is what pins the context length; "
+ "before it, this check overflows a window nobody set"
+ )
+
+
+def test_the_over_length_case_must_report_dropped_messages():
+ func = _func("assert_compaction")
+ tests = [ast.unparse(n.test) for n in ast.walk(func) if isinstance(n, ast.If)]
+ assert any(
+ "long_dropped" in t for t in tests
+ ), "nothing depends on the long conversation having been shortened"
+
+
+def test_the_short_control_is_present_and_is_the_opposite_claim():
+ """Without it, a server that always claims truncation passes."""
+ func = _func("assert_compaction")
+ short = [
+ n
+ for n in ast.walk(func)
+ if isinstance(n, ast.If) and "short_dropped" in ast.unparse(n.test)
+ ]
+ assert short, "there is no negative control"
+ for node in short:
+ # It must fail when the short chat DID drop, which is the opposite
+ # polarity to the long one.
+ assert not (
+ isinstance(node.test, ast.UnaryOp) and isinstance(node.test.op, ast.Not)
+ ), "the control must fire when a SHORT conversation reports a drop"
+
+
+def test_a_refusal_is_a_failure_rather_than_a_pass():
+ """A context-length error is not compaction. The distinction is the whole
+ feature: one shortens the prompt and answers, the other gives up.
+
+ Asserted structurally. The first version of this test matched the failure
+ MESSAGE, with an `or` that was true of any body at all -- a guard satisfied
+ by its own surrounding text, which is the exact shape this directory has
+ been caught by five times.
+ """
+ func = _func("assert_compaction")
+ refusals = [
+ n
+ for n in ast.walk(func)
+ if isinstance(n, ast.If)
+ and ast.unparse(n.test) == "code != 200"
+ and any(
+ isinstance(c, ast.Call)
+ and isinstance(c.func, ast.Attribute)
+ and c.func.attr == "append"
+ for c in ast.walk(n)
+ )
+ ]
+ assert len(refusals) >= 2, (
+ "both the long conversation and the short control must fail on a "
+ "non-200, or a server that refuses everything reads as a pass"
+ )
+
+
+def test_the_overflow_is_built_past_the_budget_and_not_merely_to_it():
+ """The budget subtracts the reply reserve and the template's own framing,
+ so a prompt equal to the context length is not reliably over it."""
+ body = _body()
+ assert "for i in range(40):" in body
+ assert "self.args.studio_ctx" in body
+
+
+def test_the_cpu_fallback_records_the_assertion_rather_than_omitting_it():
+ assert '"compaction",' in _body("execute")
+
+
+def test_every_status_read_comes_from_a_real_call():
+ """Found by mutation, not by reasoning: assigning `code = 200` after the
+ request survived every other rule here, because they all read the shape of
+ the branches and none of them asked where the value came from.
+
+ A hardcoded status is unlikely as drift and trivial as a "fix" for a red,
+ which is the same thing.
+ """
+ func = _func("assert_compaction")
+ for node in ast.walk(func):
+ if not isinstance(node, ast.Assign):
+ continue
+ targets = {t.id for t in node.targets if isinstance(t, ast.Name)}
+ if not ({"code"} & targets or {"code", "body"} <= _tuple_names(node)):
+ continue
+ assert isinstance(node.value, ast.Call), (
+ f"`code` is assigned from {ast.unparse(node.value)!r} rather than "
+ f"from a request, so the status this checks is not the server's"
+ )
+
+
+def _tuple_names(node: ast.Assign) -> set:
+ out = set()
+ for target in node.targets:
+ if isinstance(target, ast.Tuple):
+ out.update(e.id for e in target.elts if isinstance(e, ast.Name))
+ return out
+
+
+def test_compaction_is_REQUESTED_rather_than_expected_by_default():
+ """Measured on kernel unsloth-probe-studio-full2-815a0c, where this
+ assertion failed a documented default.
+
+ `context_overflow` defaults to "error": an over-length conversation comes
+ back 400 with code=context_length_exceeded, so a client's own trim loop can
+ see it. Compaction is a policy you ASK for. "truncate_oldest" is the one
+ that applies to a plain chat; "truncate_middle" is limited to client-tool
+ and response_format passthrough
+ (studio/backend/models/inference.py:context_overflow).
+ """
+ body = _body()
+ assert 'context_overflow = "truncate_oldest"' in body
+ assert (
+ 'context_overflow = "truncate_middle"' not in body
+ ), "truncate_middle does not apply to a plain chat, so it would report a policy that never ran"
+
+
+def test_the_default_policy_control_is_present_and_expects_a_refusal():
+ """The half that stops the request field from being decorative.
+
+ Without it, the check above passes on a server that compacts everything
+ regardless of what was asked for, and naming the policy proves nothing.
+ """
+ func = _func("assert_compaction")
+ refusals = [
+ node
+ for node in ast.walk(func)
+ if isinstance(node, ast.Compare) and "400" in ast.unparse(node)
+ ]
+ assert refusals, "nothing asserts the default policy still refuses"
+ assert "long_status_default_policy" in _body()
+
+
+def test_the_refusal_is_checked_by_CODE_and_not_by_the_status_alone():
+ """400 is not the claim. `openai_error_body` always emits the `code` key and
+ leaves it None for an unclassified failure, so a validation error and a
+ code-less generation error are both 400 -- and a status-only check accepts
+ either as proof of overflow semantics the failure text promises.
+
+ `code=context_length_exceeded` is what a client's trim loop detects
+ (routes/inference.py, `_openai_stream_error_chunk`), so it is the assertion.
+ """
+ func = _func("assert_compaction")
+ body = ast.unparse(func)
+ # `ast.unparse` normalises quoting, so match the value not the literal.
+ assert (
+ "context_length_exceeded" in body
+ ), "the default-policy control never reads the error code, so any 400 passes"
+ compares = [
+ ast.unparse(node)
+ for node in ast.walk(func)
+ if isinstance(node, ast.Compare) and "context_length_exceeded" in ast.unparse(node)
+ ]
+ assert compares, "context_length_exceeded appears but is never compared against"
+ # And it has to gate the SAME branch as the status, or a run can refuse with
+ # the wrong code and still pass.
+ guards = [
+ ast.unparse(node.test)
+ for node in ast.walk(func)
+ if isinstance(node, ast.If) and "context_length_exceeded" in ast.unparse(node.test)
+ ]
+ assert any("400" in guard for guard in guards), (
+ "the code and the status are checked in separate branches, so one can "
+ "pass while the other is wrong"
+ )
+
+
+def test_the_over_length_conversation_is_sent_twice_under_two_policies():
+ """One request cannot compare two policies. Both calls have to exist, and
+ they have to send the SAME messages, or the pair compares a policy against
+ a different conversation."""
+ func = _func("assert_compaction")
+ calls = [
+ node
+ for node in ast.walk(func)
+ if isinstance(node, ast.Call)
+ and isinstance(node.func, ast.Attribute)
+ and node.func.attr == "chat"
+ and "long_messages" in ast.unparse(node)
+ ]
+ assert len(calls) == 2, f"expected two long-conversation calls, found {len(calls)}"
+ overflow = [call for call in calls if any(kw.arg == "context_overflow" for kw in call.keywords)]
+ assert len(overflow) == 1, (
+ "exactly one of the two must name the policy: both naming it removes "
+ "the control, neither naming it removes the feature under test"
+ )
diff --git a/tests/kaggle/test_studio_gpu_harness.py b/tests/kaggle/test_studio_gpu_harness.py
index 72bcf14fa21..2b81a86afa6 100644
--- a/tests/kaggle/test_studio_gpu_harness.py
+++ b/tests/kaggle/test_studio_gpu_harness.py
@@ -19,7 +19,6 @@ Whichever ran first would win.
from __future__ import annotations
-import argparse
import base64
import importlib.util
import io
@@ -994,11 +993,28 @@ def test_the_gate_and_launcher_are_the_shared_ones():
# ------------------------------------------------------------------ report
+_PASSING = [{"label": "studio-gpu", "passed": True, "assertions": []}]
+_FAILING = [{"label": "studio-gpu", "passed": False, "failures": ["x"], "assertions": []}]
+# A training leg from the merged kernel. Not this reporter's payload.
+_LEG = [{"label": "control", "passed": False, "steps": []}]
+
+
@pytest.mark.parametrize(
- ("verdict", "expected_exit"),
- [("pass", 0), ("partial", 0), ("infra", 0), ("fail", 1)],
+ ("verdict", "reports", "expected_exit"),
+ [
+ ("pass", _PASSING, 0),
+ ("partial", [], 0),
+ ("infra", [], 0),
+ ("fail", _FAILING, 1),
+ # The kernel verdict is the WHOLE kernel's, and since --with-studio
+ # that kernel also carries four training legs. A leg failing must not
+ # print "Studio GPU smoke: FAIL" over a passing Studio payload and send
+ # someone to read the wrong half: the T4 reporter is what turns that
+ # red, in the same job, from the same evidence directory.
+ ("fail", _PASSING + _LEG, 0),
+ ],
)
-def test_only_a_real_assertion_failure_turns_the_job_red(tmp_path, verdict, expected_exit):
+def test_only_a_real_assertion_failure_turns_the_job_red(tmp_path, verdict, reports, expected_exit):
evidence = tmp_path / "evidence"
evidence.mkdir()
(evidence / "launch_result.json").write_text(
@@ -1008,7 +1024,7 @@ def test_only_a_real_assertion_failure_turns_the_job_red(tmp_path, verdict, expe
"reason": "test",
"slug": "u/s",
"kernel_state": "COMPLETE",
- "reports": [],
+ "reports": reports,
}
)
)
@@ -1225,11 +1241,22 @@ def _load_payload():
def _session(module, tmp_path, **overrides):
- args = argparse.Namespace(
- repo_root = str(tmp_path / "repo"),
- studio_home = str(tmp_path / "home"),
- **overrides,
+ # Built from the REAL parser rather than a hand-listed Namespace. A
+ # hand-listed one carries exactly the attributes someone remembered, so
+ # every new flag breaks these tests with an AttributeError that says
+ # nothing about the flag -- which is what --studio-password did.
+ args = module.parse_args(
+ [
+ "--outdir",
+ str(tmp_path / "out"),
+ "--repo-root",
+ str(tmp_path / "repo"),
+ "--studio-home",
+ str(tmp_path / "home"),
+ ]
)
+ for key, value in overrides.items():
+ setattr(args, key, value)
session = module.Payload.__new__(module.Payload)
session.repo_root = Path(args.repo_root)
session.studio_home = Path(args.studio_home)
@@ -2334,3 +2361,80 @@ def test_the_kaggle_client_is_new_enough_to_read_the_only_credential_we_have():
assert pins, "no pinned kaggle client in the workflow"
assert len(set(pins)) == 1, f"jobs disagree on the kaggle client: {pins}"
assert packaging_version.Version(pins[0]) >= packaging_version.Version("2.2.0"), pins[0]
+
+
+def test_no_studio_assertion_is_wired_to_a_constant_branch():
+ """A repo-wide version of the guard that caught five vacuous guards at once.
+
+ Every rule in the Studio payload was, at some point, protected by a test
+ that asserted "the failure message appears in the source". That is
+ satisfied by `if False:` sitting above an untouched message, so disabling a
+ rule outright left its test green. Five mutations survived that way in one
+ sitting.
+
+ A constant test in an `assert_*` method means a branch that can never be
+ taken, or one always taken. Neither is something a check has any business
+ doing, and this catches it for every assertion at once rather than one test
+ at a time.
+ """
+ import ast
+
+ src = (
+ Path(__file__).resolve().parents[2]
+ / "tests"
+ / "kaggle"
+ / "studio_gpu"
+ / "run_studio_gpu.py"
+ ).read_text(encoding = "utf-8")
+ offenders = []
+ for func in ast.walk(ast.parse(src)):
+ if not (isinstance(func, ast.FunctionDef) and func.name.startswith("assert_")):
+ continue
+ for node in ast.walk(func):
+ if isinstance(node, ast.If) and isinstance(node.test, ast.Constant):
+ offenders.append(f"{func.name}: if {ast.unparse(node.test)}")
+ assert offenders == [], f"assertions wired to a constant: {offenders}"
+
+
+def test_every_assertion_carries_its_own_wall_clock():
+ """Studio is the longest payload in the kernel and had no breakdown.
+
+ On unsloth-probe-full-concurrent-417238 `studio_test` ran 1487.5s -- the
+ single largest item on the critical path -- and the report carried 19
+ assertions with no timing on any of them, so "where does Studio's time go"
+ could only be answered by buying another session.
+
+ Driven rather than grepped: a rule that only checked for a `time.time()`
+ call passes on a payload that computes the number and drops it.
+ """
+ import importlib.util
+ import types
+
+ payload = PAYLOAD_DIR / "run_studio_gpu.py"
+ spec = importlib.util.spec_from_file_location("_studio_payload_timing", payload)
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+
+ runner = types.SimpleNamespace(
+ assertions = [],
+ failures = [],
+ started = module.time.time() - 5.0,
+ record = None,
+ )
+ record = module.Payload.record.__get__(runner, module.Payload)
+ module.log = lambda *a, **k: None
+ record("first", True, {})
+ record("second", True, {"failures": []})
+
+ names = [a["name"] for a in runner.assertions]
+ assert names == ["first", "second"]
+ # The first entry measures from process start, which is the setup before
+ # any assertion -- 5s here -- and is exactly the slice that would otherwise
+ # be invisible.
+ assert runner.assertions[0]["seconds_since_previous"] >= 5.0
+ # The second measures from the first, not from the start, or every entry
+ # would read as the whole run so far and the breakdown would be useless.
+ assert runner.assertions[1]["seconds_since_previous"] < 1.0
+ # And an absolute position, so a reader can line the report up against the
+ # driver's own interval for the payload.
+ assert runner.assertions[1]["at_seconds"] >= runner.assertions[0]["at_seconds"]
diff --git a/tests/kaggle/test_studio_image_generation.py b/tests/kaggle/test_studio_image_generation.py
new file mode 100644
index 00000000000..47c450dab77
--- /dev/null
+++ b/tests/kaggle/test_studio_image_generation.py
@@ -0,0 +1,194 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""Image generation: the smallest run that can still be wrong in a visible way.
+
+256x256 at 2 steps, because the claim is that the path executes rather than
+that the picture is good.
+
+"Nothing errored" is not the check, and that is the whole design. A diffusion
+pipeline that fails part-way still writes a gallery record and still answers
+200; a pipeline whose weights never loaded produces a FLAT frame, which is a
+perfectly valid PNG. So the verdict is read off the downloaded file:
+
+* the PNG magic, so the download endpoint is serving an image rather than a
+ JSON error with a 200 on it;
+* the size out of the IHDR chunk, not out of the gallery record -- the record
+ repeats what was ASKED for and the file says what was MADE;
+* not-one-flat-colour, on decoded extrema where PIL is available and on a
+ compressed-size floor where it is not.
+"""
+
+from __future__ import annotations
+
+import ast
+import struct
+import zlib
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"
+SRC = PAYLOAD.read_text(encoding = "utf-8")
+
+
+def _func(name: str) -> ast.FunctionDef:
+ for cls in ast.walk(ast.parse(SRC)):
+ if not isinstance(cls, ast.ClassDef):
+ continue
+ for node in cls.body:
+ if isinstance(node, ast.FunctionDef) and node.name == name:
+ return node
+ raise AssertionError(f"no method named {name!r}")
+
+
+def _body(name: str = "assert_image_generation") -> str:
+ return ast.get_source_segment(SRC, _func(name)) or ""
+
+
+def _png(width: int, height: int, payload: bytes) -> bytes:
+ def chunk(kind: bytes, data: bytes) -> bytes:
+ return (
+ struct.pack(">I", len(data))
+ + kind
+ + data
+ + struct.pack(">I", zlib.crc32(kind + data) & 0xFFFFFFFF)
+ )
+
+ header = struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)
+ return b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", header) + chunk(b"IDAT", payload)
+
+
+def test_the_assertion_exists_and_is_off_by_default():
+ """Last priority, and it pulls a diffusion checkpoint the rest of the
+ payload has no use for. A dispatch that wants it says so."""
+ assert _body()
+ assert "self.assert_image_generation()" in _body("execute")
+ assert '"--image-generation",' in SRC
+ assert "self.args.image_generation" in _body("execute")
+
+
+def test_the_run_is_the_smallest_the_schema_allows():
+ body = _body()
+ assert "want = 256" in body, "256 is the schema's floor for width and height"
+ assert '"steps": 2,' in body
+
+
+def test_the_size_is_read_from_the_file_and_not_from_the_record():
+ """The gallery record repeats what was asked for. Comparing the request
+ against itself is the vacuity this rule exists against."""
+ body = _body()
+ assert 'int.from_bytes(png[16:20], "big")' in body
+ assert 'int.from_bytes(png[20:24], "big")' in body
+
+
+def test_the_ihdr_offsets_are_right():
+ """Executed against a PNG built here, because an off-by-four in a
+ fixed-offset parse reads a plausible number out of the wrong bytes and
+ every other rule still passes."""
+ blob = _png(256, 256, zlib.compress(b"\x00" * 16))
+ assert int.from_bytes(blob[16:20], "big") == 256
+ assert int.from_bytes(blob[20:24], "big") == 256
+ wrong = _png(512, 64, zlib.compress(b"\x00" * 16))
+ assert int.from_bytes(wrong[16:20], "big") == 512
+ assert int.from_bytes(wrong[20:24], "big") == 64
+
+
+def test_a_flat_image_is_a_failure():
+ """A pipeline whose weights never loaded returns a uniform frame, and a
+ uniform frame is a valid PNG of exactly the right size. Every other rule
+ here passes on it."""
+ func = _func("assert_image_generation")
+ assert any(
+ isinstance(n, ast.If) and ast.unparse(n.test) == "flat" for n in ast.walk(func)
+ ), "nothing fails on a flat image"
+ body = _body()
+ assert "getextrema()" in body
+ assert "flatness_source" in body, "the reader must be able to see which rule ruled"
+
+
+def test_the_png_magic_is_checked():
+ body = _body()
+ assert 'png.startswith(b"\\x89PNG\\r\\n\\x1a\\n")' in body
+
+
+def test_the_bytes_are_fetched_raw_rather_than_through_the_json_client():
+ """`Studio.request` decodes to utf-8, which corrupts the bytes this whole
+ assertion is about."""
+ body = _body()
+ assert "urllib.request.urlopen" in body
+ assert "self.studio.get(" not in body.split("gallery/{image_id}/file")[0][-400:]
+
+
+def test_the_pipeline_is_unloaded_in_a_finally():
+ """A diffusion pipeline is the largest single thing this payload puts on a
+ T4. Left resident, it takes the card from whatever runs next and that
+ failure lands on the wrong assertion."""
+ func = _func("assert_image_generation")
+ finals = "\n".join(
+ ast.unparse(n) for t in ast.walk(func) if isinstance(t, ast.Try) for n in t.finalbody
+ )
+ assert "images/unload" in finals
+
+
+def test_a_load_or_generate_error_is_a_failure_rather_than_a_skip():
+ body = _body()
+ assert 'failures.append(f"images/load returned HTTP' in body
+ assert "failures.append(" in body
+ func = _func("assert_image_generation")
+ assert any(
+ isinstance(n, ast.If) and "code >= 400" in ast.unparse(n.test) for n in ast.walk(func)
+ )
+
+
+def test_it_runs_while_the_server_is_still_up():
+ """`assert_chat_ui` ends by clicking Stop server and asserting the port
+ closes, so every request after it is refused at the socket.
+
+ On kernel unsloth-probe-studio-full2-815a0c this assertion reported
+ `URLError: Connection refused` and read as a broken image path on a server
+ that had simply been shut down. Ordering, not the image pipeline.
+ """
+ run = _body("execute")
+ image_at = run.index("self.assert_image_generation()")
+ ui_at = run.index("self.assert_chat_ui()")
+ assert image_at < ui_at, (
+ "image generation is driven after the UI driver stops the server, so "
+ "it can only ever report a connection error"
+ )
+
+
+def test_the_load_is_WAITED_ON_rather_than_assumed_synchronous():
+ """`images/load` answers 200 having only ACCEPTED the request.
+
+ On kernel unsloth-probe-studio-r3-0b85d4 `load_status` was 200 and
+ `generate_status` was 409 with "No diffusion model is loaded." -- an
+ assertion failing on its own impatience. The wait reads `images/status`
+ for `loaded`, and carries `images/load-progress` alongside so a download
+ that stalls or errors is reported as that rather than as a broken
+ generation.
+ """
+ body = _body()
+ assert "/api/inference/images/status" in body
+ assert "/api/inference/images/load-progress" in body
+ assert 'status.get("loaded")' in body
+
+ func = _func("assert_image_generation")
+ src = ast.get_source_segment(SRC, func) or ""
+ wait_at = src.index('status.get("loaded")')
+ generate_at = src.index("/api/inference/images/generate")
+ assert wait_at < generate_at, "the wait must come before the generate call"
+
+
+def test_a_load_that_never_finishes_fails_rather_than_generating_anyway():
+ """The refusal branch. Falling through to `generate` after the deadline
+ reports a 409 under the generation's name, which sends the reader after the
+ wrong component -- which is exactly what happened."""
+ func = _func("assert_image_generation")
+ src = ast.get_source_segment(SRC, func) or ""
+ marker = "never reported loaded within"
+ assert marker in src
+ after = src[src.index(marker) :]
+ assert "return self.record" in after.split("/api/inference/images/generate")[0], (
+ "the deadline branch does not return, so it generates against a model "
+ "that was never loaded"
+ )
diff --git a/tests/kaggle/test_studio_lora_vs_base.py b/tests/kaggle/test_studio_lora_vs_base.py
new file mode 100644
index 00000000000..6165b142559
--- /dev/null
+++ b/tests/kaggle/test_studio_lora_vs_base.py
@@ -0,0 +1,174 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""The exported model against the base it came from.
+
+`assert_gguf_export` proves a file was produced, carries the GGUF magic, loads
+on the GPU and generates. Every one of those is ALSO true of an export that
+silently merged nothing and shipped the base weights, which is the regression
+worth catching and is invisible to file size, to the magic, and to "it
+generated text".
+
+Greedy decoding at temperature 0 makes it visible: identical weights answer
+identically, so a difference is the adapter.
+
+The determinism control is the part that makes that argument valid, and it runs
+FIRST. If the same weights loaded twice do not reproduce their own answer, a
+difference between two models is noise, and the assertion has to say it could
+not compare rather than pass on it.
+
+What is deliberately NOT asserted is the canary. Studio's training run is a
+handful of steps, and whether that is enough to learn a specific string is a
+property of the run length rather than of the export path: asserting it would
+be a red about training tuning wearing an export label.
+"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"
+SRC = PAYLOAD.read_text(encoding = "utf-8")
+
+
+def _func(name: str) -> ast.FunctionDef:
+ for cls in ast.walk(ast.parse(SRC)):
+ if not isinstance(cls, ast.ClassDef):
+ continue
+ for node in cls.body:
+ if isinstance(node, ast.FunctionDef) and node.name == name:
+ return node
+ raise AssertionError(f"no method named {name!r}")
+
+
+def _body(name: str = "assert_lora_vs_base") -> str:
+ return ast.get_source_segment(SRC, _func(name)) or ""
+
+
+def test_the_assertion_exists_and_is_driven_from_the_run():
+ assert _body()
+ assert "self.assert_lora_vs_base(" in _body("execute")
+
+
+def test_it_compares_against_the_file_the_export_actually_made():
+ """A path recomputed here could name a stale GGUF from an earlier run,
+ which would compare the base against something this session never made."""
+ body = _body("execute")
+ assert 'entry["name"] == "gguf_export"' in body
+ assert 'exported_gguf = entry.get("gguf")' in body
+
+
+def test_the_claim_is_that_the_two_differ():
+ func = _func("assert_lora_vs_base")
+ guarded = [
+ n
+ for n in ast.walk(func)
+ if isinstance(n, ast.If) and "tuned == base_one" in ast.unparse(n.test)
+ ]
+ assert guarded, (
+ "nothing fails when the export answers exactly as the base does, "
+ "which is what a no-op merge looks like"
+ )
+
+
+def test_the_determinism_control_runs_before_the_claim():
+ """Ordering is the argument. A difference only means "adapter" once the
+ same weights have been shown to reproduce themselves."""
+ body = _body()
+ control_at = body.index("base_one != base_two")
+ claim_at = body.index("tuned == base_one")
+ assert control_at < claim_at
+
+
+def test_a_non_reproducing_base_is_reported_as_a_failure_not_a_pass():
+ """ "I could not compare" and "they matched" are opposite outcomes. Letting
+ the first through as a pass is the exact shape this directory keeps being
+ caught by."""
+ func = _func("assert_lora_vs_base")
+ for node in ast.walk(func):
+ if isinstance(node, ast.If) and "base_one != base_two" in ast.unparse(node.test):
+ appends = [
+ n
+ for n in ast.walk(node)
+ if isinstance(n, ast.Call)
+ and isinstance(n.func, ast.Attribute)
+ and n.func.attr == "append"
+ ]
+ assert appends, "a base that will not reproduce itself must fail"
+ break
+ else:
+ raise AssertionError("there is no determinism control at all")
+
+
+def test_the_canary_is_recorded_and_never_asserted():
+ """It is a property of how many steps the training ran, not of the export
+ path. Failing on it would put a training-tuning red under an export name."""
+ func = _func("assert_lora_vs_base")
+ for node in ast.walk(func):
+ if isinstance(node, ast.If) and "CANARY" in ast.unparse(node.test):
+ raise AssertionError(
+ "the verdict branches on the canary, which makes this red "
+ "whenever the training run is too short to learn it"
+ )
+ assert "canary_in_exported" in _body()
+
+
+def test_a_missing_export_is_a_failure_rather_than_a_skip():
+ body = _body()
+ assert "if not gguf:" in body
+ assert 'return self.record("lora_vs_base", False, detail)' in body
+
+
+def test_both_models_are_driven_with_the_same_prompt():
+ """Two different questions would differ for reasons that have nothing to
+ do with the adapter."""
+ func = _func("assert_lora_vs_base")
+ prompts = [
+ n
+ for n in ast.walk(func)
+ if isinstance(n, ast.Assign)
+ and any(isinstance(t, ast.Name) and t.id == "prompt" for t in n.targets)
+ ]
+ assert len(prompts) == 1, "the prompt must be built once and reused"
+ assert (
+ _body().count("self.chat(prompt)") == 1
+ ), "one call site, reached by every arm, or the arms can drift apart"
+
+
+def test_the_exported_gguf_picked_is_a_model_and_not_an_mmproj_sidecar():
+ """Driven through the REAL `newest_gguf`, because this is a selection bug
+ and a rule fed a path proves nothing about the selector.
+
+ A vision export writes two files, and the projector is often the newer.
+ Handing `Qwen3.5-2B.F16-mmproj.gguf` to llama.cpp as a model is not an
+ error: the server starts, reports gpu_layers=-1, offloads nothing and still
+ returns text. On kernel unsloth-probe-studio-full2-815a0c that failed both
+ the export assertion and this one, as a GPU fallback that never happened.
+ """
+ import sys as _sys # noqa: PLC0415
+ import tempfile # noqa: PLC0415
+ import time as _time # noqa: PLC0415
+ from pathlib import Path as _Path # noqa: PLC0415
+
+ _sys.path.insert(0, str(ROOT / "tests" / "kaggle" / "studio_gpu"))
+ from studio_client import newest_gguf # noqa: PLC0415
+
+ with tempfile.TemporaryDirectory() as tmp:
+ root = _Path(tmp)
+ model = root / "Qwen3.5-2B.Q8_0.gguf"
+ model.write_bytes(b"GGUF")
+ _time.sleep(0.01)
+ # Written LAST, so "newest" alone would pick it.
+ sidecar = root / "Qwen3.5-2B.F16-mmproj.gguf"
+ sidecar.write_bytes(b"GGUF")
+
+ assert newest_gguf(root) == model
+
+ # And with only a sidecar present, the answer is None rather than the
+ # sidecar: no model was exported, and saying so is the point.
+ sidecar_only = root / "sub"
+ sidecar_only.mkdir()
+ (sidecar_only / "x.F16-mmproj.gguf").write_bytes(b"GGUF")
+ assert newest_gguf(sidecar_only) is None
diff --git a/tests/kaggle/test_studio_mtp_engages.py b/tests/kaggle/test_studio_mtp_engages.py
new file mode 100644
index 00000000000..8be55969d9d
--- /dev/null
+++ b/tests/kaggle/test_studio_mtp_engages.py
@@ -0,0 +1,96 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""Selecting the MTP repo is a REQUEST; the drafter engaging is the result.
+
+`--chat-model unsloth/Qwen3.5-2B-MTP-GGUF` was chosen because multi-token
+prediction is a distinct serving path in llama.cpp. But when the companion is
+missing, the binary was built without MTP, or the drafter is downgraded for
+VRAM, the main GGUF still loads and still generates -- so every inference
+assertion stays green while the path the repo was chosen for never ran. That is
+the shape this directory keeps being caught by: nothing red, and a claim nobody
+made.
+
+`spec_drafter_kind` is what settles it, and `spec_fallback_reason` names WHICH
+of those happened, since "llama.cpp has no MTP" and "the drafter was downgraded
+for VRAM" are different findings and only one is about this leg.
+"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"
+SRC = PAYLOAD.read_text(encoding = "utf-8")
+TREE = ast.parse(SRC)
+
+
+def _func(name: str) -> ast.FunctionDef:
+ for node in ast.walk(TREE):
+ if isinstance(node, ast.FunctionDef) and node.name == name:
+ return node
+ raise AssertionError(f"{name} is gone")
+
+
+def test_the_default_chat_model_is_the_mtp_repo():
+ """If this stops being an MTP repo the assertion below becomes vacuous, so
+ it fails here rather than passing silently."""
+ assert "unsloth/Qwen3.5-2B-MTP-GGUF" in SRC
+
+
+def test_the_status_fields_are_read_from_the_load():
+ body = ast.unparse(_func("load_model"))
+ for field in ("spec_drafter_kind", "spec_fallback_reason", "llama_cpp_supports_mtp"):
+ assert field in body, f"{field} is never read, so the drafter cannot be checked"
+
+
+def test_a_drafter_that_never_engaged_is_a_failure():
+ """The claim itself. Without this, an MTP repo whose drafter fell back
+ serves ordinary decoding and reports success."""
+ func = _func("assert_gpu_inference")
+ body = ast.unparse(func)
+ assert "drafter_kind" in body, "the drafter kind is never compared"
+ guards = [
+ ast.unparse(node.test)
+ for node in ast.walk(func)
+ if isinstance(node, ast.If) and "drafter_kind" in ast.unparse(node.test)
+ ]
+ assert guards, "spec_drafter_kind appears but gates nothing"
+ assert any(
+ "'mtp'" in guard or '"mtp"' in guard for guard in guards
+ ), "the drafter kind is read but never required to be mtp"
+ appends = [
+ ast.unparse(node)
+ for node in ast.walk(func)
+ if isinstance(node, ast.Call)
+ and "failures" in ast.unparse(node)
+ and "append" in ast.unparse(node)
+ and "drafter" in ast.unparse(node)
+ ]
+ assert appends, "a fallback is noticed but not recorded as a failure"
+
+
+def test_the_check_is_scoped_to_an_mtp_model():
+ """A leg pointed at a plain GGUF must not go red for not doing MTP: that is
+ the correct behaviour there, and a rule that cannot pass is deleted."""
+ func = _func("assert_gpu_inference")
+ guards = [
+ ast.unparse(node.test)
+ for node in ast.walk(func)
+ if isinstance(node, ast.If) and "drafter_kind" in ast.unparse(node.test)
+ ]
+ assert any("MTP" in guard and "chat_model" in guard for guard in guards), (
+ "the MTP requirement is unconditional, so a plain-GGUF run fails on "
+ "behaviour that is correct for it"
+ )
+
+
+def test_the_reason_is_reported_and_not_just_the_verdict():
+ """'MTP is off' sends the reader nowhere. The reason distinguishes a build
+ without MTP from a VRAM downgrade, and only one of those is actionable."""
+ func = _func("assert_gpu_inference")
+ body = ast.unparse(func)
+ assert "fallback_reason" in body
+ assert "supports_mtp" in body
diff --git a/tests/kaggle/test_studio_password_path.py b/tests/kaggle/test_studio_password_path.py
new file mode 100644
index 00000000000..273bc8f623e
--- /dev/null
+++ b/tests/kaggle/test_studio_password_path.py
@@ -0,0 +1,107 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""Starting Studio headless with --password, and the way that check goes vacuous.
+
+`unsloth studio --password X` sets the INITIAL admin password when none is set
+yet, which is exactly the state a scripted server is in. Without it the only way
+in is the bootstrap password Studio seeds into a file and prints to its own log,
+and a deployment that has to read a log to log in is not one anybody scripts
+twice.
+
+**The vacuity to avoid is a fallback.** If logging in with the passed password
+failed and the payload then tried the bootstrap, the run would go green while
+proving `--password` does nothing at all. There is deliberately no fallback, and
+that is asserted from the source here.
+
+**The second risk is a leaked credential.** The password reaches a process
+command line, Studio's startup banner, `studio.log`, and the evidence bundle
+that travels home. It is generated per run and registered as a secret before
+anything can log it, so every log leaving the machine is scrubbed.
+"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"
+SRC = PAYLOAD.read_text(encoding = "utf-8")
+
+
+def test_the_password_reaches_the_studio_command():
+ assert 'cmd += ["--password", self.args.studio_password]' in SRC
+
+
+def test_login_uses_the_password_that_was_passed():
+ assert "self.studio.login(self.args.studio_password)" in SRC
+
+
+def test_there_is_no_fallback_to_the_bootstrap_password():
+ """The whole assertion. If --password were ignored, Studio would seed a
+ bootstrap password instead and the login would fail; a fallback would then
+ quietly succeed and report a pass for a flag that did nothing."""
+ tree = ast.parse(SRC)
+ func = next(
+ node
+ for node in ast.walk(tree)
+ if isinstance(node, ast.FunctionDef) and node.name == "authenticate"
+ )
+ # Find the `if self.args.studio_password:` branch and assert it RETURNS
+ # rather than falling through into the bootstrap path below it.
+ branch = next(
+ node
+ for node in func.body
+ if isinstance(node, ast.If) and "studio_password" in ast.dump(node.test)
+ )
+ # The LAST statement of the branch, unconditionally, and not merely "a
+ # Return somewhere inside". `if not failures: return ...` contains a Return
+ # and still falls through on the failure path -- which is the single case
+ # this guard exists for. That mutation survived the first version of this
+ # assertion, so the rule is now about the branch always returning.
+ assert isinstance(branch.body[-1], ast.Return), (
+ "the --password branch must END in an unconditional return; a "
+ "conditional one falls through to the bootstrap path on failure and "
+ "turns a flag that did nothing into a pass"
+ )
+ branch_src = ast.get_source_segment(SRC, branch) or ""
+ assert "remember_bootstrap" not in branch_src, "the --password branch reads the bootstrap"
+
+
+def test_the_generated_password_is_registered_as_a_secret_before_use():
+ """Registered in __init__, which runs before the server starts and before
+ anything writes a log. Registering it later would scrub the logs written
+ after that point and not the banner."""
+ tree = ast.parse(SRC)
+ init = next(
+ node
+ for node in ast.walk(tree)
+ if isinstance(node, ast.FunctionDef) and node.name == "__init__"
+ )
+ body = ast.get_source_segment(SRC, init) or ""
+ assert "self.secrets.add(self.args.studio_password)" in body
+ assert "secrets_module.token_urlsafe" in body, "auto must mint a fresh value per run"
+
+
+def test_no_constant_password_is_committed():
+ """A constant in a repo is a credential whether or not it is reachable."""
+ tree = ast.parse(SRC)
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Call) and getattr(node.func, "attr", "") == "add_argument":
+ names = [a.value for a in node.args if isinstance(a, ast.Constant)]
+ if "--studio-password" in names:
+ for kw in node.keywords:
+ if kw.arg == "default":
+ assert (
+ kw.value.value == ""
+ ), f"a default password is committed: {kw.value.value!r}"
+ break
+ else:
+ raise AssertionError("--studio-password is not declared at all")
+
+
+def test_the_default_keeps_the_previous_bootstrap_behaviour():
+ """Empty means "use the bootstrap", so a caller that does not opt in is
+ unaffected and the existing Studio leg does not change shape."""
+ assert '"--studio-password",\n default = "",' in SRC
diff --git a/tests/kaggle/test_studio_server_flags.py b/tests/kaggle/test_studio_server_flags.py
new file mode 100644
index 00000000000..785a56b0b87
--- /dev/null
+++ b/tests/kaggle/test_studio_server_flags.py
@@ -0,0 +1,283 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""llama.cpp server flags: a quantized KV cache, a two-card split, a pinned ctx.
+
+The vacuity here is specific and easy to write by accident: **asserting that
+the load succeeded**. llama-server starts happily when a flag it does not like
+is dropped, so a load that ignored `cache_type_kv` entirely, fell back to one
+card, and ran at the model's default context is indistinguishable from a
+correct one by return code alone.
+
+Studio's status separates the request from what is in force, which is what
+makes a real check possible. So the rules are about the APPLIED values, and a
+downgrade is allowed only when Studio says why -- a model whose cache layout
+cannot be quantized is entitled to refuse, and silence is not.
+"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"
+SRC = PAYLOAD.read_text(encoding = "utf-8")
+
+
+def _func(name: str = "assert_server_flags") -> ast.FunctionDef:
+ tree = ast.parse(SRC)
+ return next(n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == name)
+
+
+def _body() -> str:
+ tree = ast.parse(SRC)
+ func = next(
+ n
+ for n in ast.walk(tree)
+ if isinstance(n, ast.FunctionDef) and n.name == "assert_server_flags"
+ )
+ return ast.get_source_segment(SRC, func) or ""
+
+
+def test_the_load_requests_all_three_flags():
+ body = _body()
+ assert '"cache_type_kv": "q8_0"' in body
+ # Sized to the VISIBLE cards rather than hardcoded to two.
+ # --studio-concurrent pins this half to one card so it can share with a
+ # training leg, and [1.0, 1.0] against a one-card server asks llama.cpp to
+ # split across a device that is not there: what comes back is a failure
+ # about the load rather than about the flag.
+ assert '"tensor_split": [1.0] * max(1, len(cards))' in body
+ assert "cards = gpu_inventory()" in body
+ assert '"max_seq_length": self.args.studio_ctx' in body
+
+
+def test_a_one_card_run_says_the_two_card_split_was_not_exercised():
+ """The coverage this shares away must be STATED, not silently dropped.
+
+ Under --studio-concurrent the split is over one device, which is not the
+ flag the brief asks about. A check that keeps its name while testing less
+ is the failure this directory keeps being caught by, so the report carries
+ `tensor_split_over_two_cards` and a note naming what was and was not
+ covered.
+ """
+ body = _body()
+ assert 'detail["tensor_split_over_two_cards"] = len(cards) >= 2' in body
+ assert "was NOT exercised" in body
+ # And it is a RECORD, not a failure: sharing is a scheduling decision, so a
+ # one-card run is not a defect and must not go red for being one.
+ func = _func("assert_server_flags")
+ for node in ast.walk(func):
+ if isinstance(node, ast.If) and "cards" in ast.unparse(node.test):
+ appended = [
+ inner
+ for inner in ast.walk(node)
+ if isinstance(inner, ast.Call)
+ and isinstance(inner.func, ast.Attribute)
+ and inner.func.attr == "append"
+ ]
+ assert not appended, (
+ "a one-card run fails rather than records, so every "
+ "--studio-concurrent run would go red by design"
+ )
+
+
+def test_the_check_reads_the_applied_values_not_the_request():
+ """The whole point. A check on the request is a check on a dict this file
+ just built."""
+ body = _body()
+ assert '"/api/inference/status"' in body
+ assert 'applied.get("cache_type_kv")' in body
+ assert 'applied.get("context_length")' in body
+
+
+def test_no_branch_in_the_check_is_wired_to_a_constant():
+ """The guard that caught five vacuous guards, including four of my own.
+
+ Every rule in this file was first written as "the failure message appears
+ in the source". That is satisfied by `if False:` above an untouched
+ message, so disabling a rule outright left the test green -- the exact
+ "assertion satisfied by its own surrounding text" failure this repo has
+ recorded before.
+
+ A constant test means a branch that can never be taken (or always is), and
+ no rule here has any business being either.
+ """
+ tree = ast.parse(_body())
+ constants = [
+ ast.unparse(node.test)
+ for node in ast.walk(tree)
+ if isinstance(node, ast.If) and isinstance(node.test, ast.Constant)
+ ]
+ assert constants == [], f"branches wired to a constant: {constants}"
+
+
+def test_the_cache_rules_append_a_failure_on_both_paths():
+ """Both the missing-value path and the silent-downgrade path must be able
+ to fail, and each is reached through its own branch."""
+ tree = ast.parse(_body())
+ messages = [
+ ast.unparse(node)
+ for node in ast.walk(tree)
+ if isinstance(node, ast.Call) and getattr(node.func, "attr", "") == "append"
+ ]
+ joined = " ".join(messages)
+ assert "no cache_type_kv at all" in joined
+ assert "silent downgrade" in joined.lower()
+
+
+def test_the_context_pin_is_checked_against_the_requested_value():
+ """llama-server admits a prompt on n_ctx alone, so a server at the model
+ default behaves differently from one at 2048 and the difference does not
+ show in a chat response."""
+ body = _body()
+ assert "int(ctx) > self.args.studio_ctx" in body
+
+
+def test_gpu_residency_is_confirmed_after_the_split():
+ """A tensor_split that fell back to CPU reports a healthy server and proves
+ nothing about either card. Checked structurally: the threshold comparison
+ must exist, not merely the message about it."""
+ tree = ast.parse(_body())
+ compares = [ast.unparse(node) for node in ast.walk(tree) if isinstance(node, ast.Compare)]
+ assert any(
+ "used" in c and "200" in c for c in compares
+ ), f"no residency threshold comparison: {compares}"
+ assert "nvidia_used_mib()" in _body()
+
+
+def test_it_is_skipped_rather_than_passed_when_the_model_is_not_on_the_gpu():
+ """Structural: the skip branch must record FALSE. Recording True would turn
+ "we could not test this" into "this passed"."""
+ tree = ast.parse(SRC)
+ records = [
+ node
+ for node in ast.walk(tree)
+ if isinstance(node, ast.Call)
+ and getattr(node.func, "attr", "") == "record"
+ and node.args
+ and isinstance(node.args[0], ast.Constant)
+ and node.args[0].value == "server_flags"
+ ]
+ skips = [n for n in records if len(n.args) > 1 and "skipped" in ast.unparse(n)]
+ assert skips, "no skip path records server_flags"
+ for node in skips:
+ assert (
+ node.args[1].value is False
+ ), "the skip path records a PASS, so an untested flag reads as a working one"
+
+
+def test_it_runs_between_gpu_inference_and_training():
+ """It reloads the chat model with different flags. Earlier would change the
+ model the inference checks measured; later would put a reload between the
+ adapter and the export."""
+ infer_at = SRC.index("gpu_ok = self.assert_gpu_inference()")
+ flags_at = SRC.index("self.assert_server_flags()")
+ train_at = SRC.index("trained = self.assert_training()")
+ assert infer_at < flags_at < train_at
+
+ # And it is guarded by gpu_ok rather than by a constant, or the call sits
+ # in the right place and never runs.
+ tree = ast.parse(SRC)
+ calls = [
+ node
+ for node in ast.walk(tree)
+ if isinstance(node, ast.If)
+ and "assert_server_flags" in ast.unparse(node)
+ and not isinstance(node, ast.Module)
+ ]
+ assert calls, "the call is not inside a branch at all"
+ assert any(ast.unparse(n.test) == "gpu_ok" for n in calls), (
+ f"the call is gated on something other than gpu_ok: "
+ f"{[ast.unparse(n.test) for n in calls]}"
+ )
+
+
+def test_the_context_default_is_the_one_the_brief_asks_for():
+ assert '"--studio-ctx",' in SRC
+ assert "default = 2048," in SRC
+
+
+def _payload_module():
+ """The real payload, imported by path so the rules DRIVE it.
+
+ Every rule above this point reads the source with `ast`, which is the right
+ instrument for "does the branch exist" and the wrong one for "does it answer
+ correctly". The bug below was invisible to all of them: the code was
+ exactly as written and the value it produced was false.
+ """
+ import importlib.util
+
+ spec = importlib.util.spec_from_file_location("_studio_payload_flags", PAYLOAD)
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+def _with_smi(module, monkeypatch, rows: str, visible: str | None):
+ class _Proc:
+ returncode = 0
+ stdout = rows
+
+ monkeypatch.setattr(module, "run", lambda *a, **k: _Proc())
+ if visible is None:
+ monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising = False)
+ else:
+ monkeypatch.setenv("CUDA_VISIBLE_DEVICES", visible)
+
+
+TWO_ROWS = "Tesla T4, 15360 MiB, 7.5\nTesla T4, 15360 MiB, 7.5\n"
+
+
+def test_a_pinned_payload_does_not_see_both_cards(monkeypatch):
+ """The exact reading from unsloth-probe-full-concurrent-417238.
+
+ nvidia-smi lists two T4s. build_kernel.py:835 pinned this payload to card 0
+ with CUDA_VISIBLE_DEVICES. The report recorded `cards_visible: 2` and
+ `tensor_split_over_two_cards: True` and sent `tensor_split: [1.0, 1.0]` to a
+ one-card server, which loaded anyway -- so the assertion passed green while
+ asking llama.cpp to split across a device that was not there.
+ """
+ module = _payload_module()
+ _with_smi(module, monkeypatch, TWO_ROWS, "0")
+ assert len(module.gpu_inventory()) == 1
+
+
+def test_an_unpinned_payload_still_sees_every_card(monkeypatch):
+ """The other direction, and it is the one the two-card claim depends on: a
+ leg that is deliberately NOT pinned must still report both cards, or real
+ multi-GPU coverage would be reported as single-card."""
+ module = _payload_module()
+ _with_smi(module, monkeypatch, TWO_ROWS, None)
+ assert len(module.gpu_inventory()) == 2
+ _with_smi(module, monkeypatch, TWO_ROWS, "0,1")
+ assert len(module.gpu_inventory()) == 2
+
+
+def test_an_empty_setting_means_no_cards_rather_than_all_of_them(monkeypatch):
+ """CUDA_VISIBLE_DEVICES="" is a deliberate "no GPU" and is not the same as
+ unset. Reading it as unset would report a full inventory on a session that
+ has no card at all, which preflight exists to catch."""
+ module = _payload_module()
+ _with_smi(module, monkeypatch, TWO_ROWS, "")
+ assert module.gpu_inventory() == []
+
+
+def test_a_uuid_selection_still_counts_the_cards(monkeypatch):
+ """CUDA_VISIBLE_DEVICES may name GPU-<uuid> rather than an index. The
+ description cannot be recovered from an nvidia-smi row, but the COUNT is
+ what tensor_split is sized from, so it must stay right."""
+ module = _payload_module()
+ _with_smi(module, monkeypatch, TWO_ROWS, "GPU-abcdef12")
+ assert len(module.gpu_inventory()) == 1
+
+
+def test_the_split_is_sized_from_the_visible_cards_and_not_from_nvidia_smi():
+ """The link between the fix and the flag. `tensor_split` is built from
+ `gpu_inventory()`, so a rule that only tested the helper would leave the
+ request free to be sized from anything."""
+ body = _body()
+ assert "cards = gpu_inventory()" in body
+ assert "[1.0] * max(1, len(cards))" in body
+ assert 'detail["tensor_split_over_two_cards"] = len(cards) >= 2' in body
diff --git a/tests/kaggle/test_studio_tabs.py b/tests/kaggle/test_studio_tabs.py
new file mode 100644
index 00000000000..2dd7e6e985a
--- /dev/null
+++ b/tests/kaggle/test_studio_tabs.py
@@ -0,0 +1,139 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""Every Studio tab's backing endpoint, checked against the routes that exist.
+
+The rest of this payload touches inference, training and export. A tab whose
+router raised on import is invisible to all of it: the route is simply never
+mounted, and the tab renders an error the moment a user opens it.
+
+This is a smoke and says so. What makes it more than a list of strings is the
+rule below that every path in `TAB_ENDPOINTS` resolves to a route actually
+declared in `studio/backend/routes`. A typo'd path 404s exactly like a missing
+router, so without that rule the payload could go red for the wrong reason --
+or, worse, a path could be quietly "fixed" to something that always answers.
+
+One endpoint choice is load-bearing: `/api/data-recipe/jobs/current` 404s by
+design when no job is running, so using it would be red on correct behaviour.
+"""
+
+from __future__ import annotations
+
+import ast
+import re
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"
+SRC = PAYLOAD.read_text(encoding = "utf-8")
+ROUTES = ROOT / "studio" / "backend" / "routes"
+MAIN = ROOT / "studio" / "backend" / "main.py"
+
+
+def _cls() -> ast.ClassDef:
+ return next(n for n in ast.walk(ast.parse(SRC)) if isinstance(n, ast.ClassDef))
+
+
+def _endpoints() -> list[tuple[str, str]]:
+ for node in _cls().body:
+ if isinstance(node, ast.Assign) and any(
+ isinstance(t, ast.Name) and t.id == "TAB_ENDPOINTS" for t in node.targets
+ ):
+ return [
+ (e.elts[0].value, e.elts[1].value)
+ for e in node.value.elts
+ if isinstance(e, ast.Tuple)
+ ]
+ raise AssertionError("TAB_ENDPOINTS is gone")
+
+
+def _func(name: str) -> ast.FunctionDef:
+ for node in _cls().body:
+ if isinstance(node, ast.FunctionDef) and node.name == name:
+ return node
+ raise AssertionError(f"no method named {name!r}")
+
+
+def _body(name: str = "assert_tabs") -> str:
+ return ast.get_source_segment(SRC, _func(name)) or ""
+
+
+def _mounted_prefixes() -> set[str]:
+ return set(
+ re.findall(r'include_router\([^)]*prefix\s*=\s*"([^"]+)"', MAIN.read_text(encoding = "utf-8"))
+ )
+
+
+def _declared_get_paths() -> set[str]:
+ paths: set[str] = set()
+ for path in ROUTES.rglob("*.py"):
+ paths.update(re.findall(r'@\w*router\.get\(\s*"([^"]+)"', path.read_text(encoding = "utf-8")))
+ return paths
+
+
+def test_the_assertion_exists_and_is_driven_from_the_run():
+ assert _body()
+ assert "self.assert_tabs()" in _body("execute")
+
+
+def test_every_endpoint_resolves_to_a_route_that_exists():
+ """The rule that turns a list of strings into a check. A typo'd path 404s
+ exactly like a missing router, so the payload would go red naming a tab
+ that is perfectly fine."""
+ prefixes = _mounted_prefixes()
+ declared = _declared_get_paths()
+ unresolved = []
+ for name, path in _endpoints():
+ for prefix in prefixes:
+ if path.startswith(prefix) and (path[len(prefix) :] or "/") in declared:
+ break
+ else:
+ unresolved.append(f"{name}: {path}")
+ assert unresolved == [], f"these paths match no declared GET route: {unresolved}"
+
+
+def test_the_four_tabs_the_directive_names_are_all_covered():
+ names = {name for name, _ in _endpoints()}
+ for wanted in ("data_designer", "image_creation", "video_creation", "image_training"):
+ assert any(n.startswith(wanted) for n in names), f"{wanted} is not covered"
+
+
+def test_it_does_not_use_an_endpoint_that_404s_by_design():
+ """`/jobs/current` raises 404 when no job is running. Using it would make
+ this red on correct behaviour, which is how a check gets switched off."""
+ assert all(path != "/api/data-recipe/jobs/current" for _, path in _endpoints())
+
+
+def test_a_404_is_distinguished_from_other_failures():
+ """A 500 is a live route with a broken handler; a 404 is a route that does
+ not exist. They send a reader to different places."""
+ func = _func("assert_tabs")
+ tests = [ast.unparse(n.test) for n in ast.walk(func) if isinstance(n, ast.If)]
+ assert any(t == "code == 404" for t in tests)
+ assert any("code >= 400" in t for t in tests)
+
+
+def test_a_200_that_is_not_a_json_object_fails():
+ """Most of these are declared with a response_model, so a bare string or a
+ null means something is standing in for the real handler."""
+ func = _func("assert_tabs")
+ assert any(
+ "isinstance(body, (dict, list))" in ast.unparse(n.test)
+ for n in ast.walk(func)
+ if isinstance(n, ast.If)
+ )
+
+
+def test_an_endpoint_that_raises_is_a_failure_rather_than_a_skip():
+ func = _func("assert_tabs")
+ handlers = [n for n in ast.walk(func) if isinstance(n, ast.ExceptHandler)]
+ assert handlers, "a transport error must not end the loop silently"
+ joined = "\n".join(ast.unparse(h) for h in handlers)
+ assert "failures.append" in joined
+
+
+def test_it_runs_before_the_long_training_phase():
+ """These need only a logged-in session. Behind a 20-minute training run, a
+ training failure hides whether the tabs exist at all."""
+ body = _body("execute")
+ assert body.index("self.assert_tabs()") < body.index("self.assert_training()")
diff --git a/tests/kaggle/test_studio_web_search.py b/tests/kaggle/test_studio_web_search.py
new file mode 100644
index 00000000000..6e285d0c903
--- /dev/null
+++ b/tests/kaggle/test_studio_web_search.py
@@ -0,0 +1,201 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""The web_search tool must be executed, and the log is the only witness.
+
+`assert_code_execution` reads its verdict off the filesystem; a web search
+leaves nothing there. What it does leave is Studio's own
+`execute_tool: name=...` line, emitted from INSIDE `execute_tool`, so it is
+written by execution rather than by selection. A loop that hands the model the
+schema and never runs the call produces no such line.
+
+Two rules stop the check drifting into vacuity in opposite directions.
+
+It must count only what THIS request wrote. The payload runs several
+tool-driven assertions against one long-lived server, and a whole-file grep
+would let `assert_code_execution`'s call satisfy this one.
+
+And it must NOT fail on an empty result set. `_web_search` fans out through
+ddgs with no API key, so a provider rate-limiting a Kaggle egress IP is a fact
+about the day rather than a Studio defect, and failing on it would put a red in
+front of every PR that no reader could act on.
+"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"
+SRC = PAYLOAD.read_text(encoding = "utf-8")
+
+
+def _func(name: str) -> ast.FunctionDef:
+ for cls in ast.walk(ast.parse(SRC)):
+ if not isinstance(cls, ast.ClassDef):
+ continue
+ for node in cls.body:
+ if isinstance(node, ast.FunctionDef) and node.name == name:
+ return node
+ raise AssertionError(f"no method named {name!r}")
+
+
+def _body(name: str = "assert_web_search") -> str:
+ return ast.get_source_segment(SRC, _func(name)) or ""
+
+
+def test_the_assertion_exists_and_is_driven_from_the_run():
+ assert _body()
+ assert "self.assert_web_search()" in _body("execute")
+
+
+def test_only_the_web_search_tool_is_offered():
+ """With `enabled_tools` omitted the loop may reach for python instead, and
+ a run that executed python would satisfy a looser log check while saying
+ nothing about search."""
+ body = _body()
+ assert "enable_tools = True" in body
+ assert 'enabled_tools = ["web_search"]' in body
+
+
+def test_the_evidence_is_the_execution_line_and_not_a_selection_one():
+ body = _body()
+ assert 'marker = "execute_tool: name=web_search"' in body, (
+ "the line must be the one execute_tool writes, because a line written "
+ "where the tool is CHOSEN is emitted whether or not it then runs"
+ )
+
+
+def test_it_counts_only_what_this_request_wrote():
+ """The payload drives several tool assertions against one server. A grep
+ over the whole log would let an earlier assertion's tool call stand in for
+ this one, which is a green tick for a search that never happened."""
+ body = _body()
+ assert "before = self.server_log.read_text" in body
+ # Whitespace-insensitive: the repo's formatter rewrites this slice to
+ # `after[len(before) :]`, and a guard that matched the unformatted spelling
+ # went red on a reformat rather than on a regression.
+ assert "fresh=after[len(before):]" in "".join(body.split())
+ assert "fresh.count(marker)" in body, "counted over the fresh slice, not the file"
+
+
+def test_an_empty_result_set_is_reported_and_not_failed():
+ """Deliberately narrow. ddgs runs with no API key, so a provider throttling
+ a Kaggle egress IP is a fact about the day; failing on it would be a red
+ nobody can act on. The execution is the claim; the results are context."""
+ func = _func("assert_web_search")
+ for node in ast.walk(func):
+ if not isinstance(node, ast.If):
+ continue
+ test = ast.unparse(node.test)
+ appends = [
+ n
+ for n in ast.walk(node)
+ if isinstance(n, ast.Call)
+ and isinstance(n.func, ast.Attribute)
+ and n.func.attr == "append"
+ ]
+ if appends and ("results" in test or "reply" in test):
+ raise AssertionError(
+ f"failing on {test!r} makes this red whenever the search "
+ f"provider is having a bad day"
+ )
+
+
+def test_the_failure_fires_when_nothing_executed():
+ func = _func("assert_web_search")
+ # Only the branches that DECIDE the verdict. The loop also breaks early on
+ # a positive count, which is a control-flow test rather than a rule, and
+ # reading it as one made this guard red on a correct body.
+ guarded = [
+ node
+ for node in ast.walk(func)
+ if isinstance(node, ast.If)
+ and "executions" in ast.unparse(node.test)
+ and any(
+ isinstance(inner, ast.Call)
+ and isinstance(inner.func, ast.Attribute)
+ and inner.func.attr == "append"
+ for inner in ast.walk(node)
+ )
+ ]
+ assert guarded, "nothing that decides the verdict depends on the tool having run"
+ assert all(
+ isinstance(n.test, ast.UnaryOp) and isinstance(n.test.op, ast.Not) for n in guarded
+ ), "the failure must fire on ZERO executions"
+
+
+def test_the_cpu_fallback_records_the_assertion_rather_than_omitting_it():
+ assert '"web_search",' in _body("execute")
+
+
+def test_the_search_tool_call_is_FORCED_rather_than_hoped_for():
+ """Otherwise this measures a model's judgement, not Studio's plumbing.
+
+ On kernel unsloth-probe-studio-full2-815a0c the 2B model answered "The
+ current version of the Linux kernel is 6.10" straight from parametric
+ knowledge, never emitted a call, and the assertion reported that Studio had
+ offered web_search and not run it. Studio had done nothing wrong.
+
+ `assert_tool_calling` already forces its own tool the same way. The claim
+ here is that Studio EXECUTES the call, and the `execute_tool` log check
+ still decides that, so forcing the call narrows the assertion onto the
+ thing it is about rather than weakening it.
+ """
+ body = _body()
+ # BY NAME, not the bare "required": that was tried on kernel
+ # unsloth-probe-studio-r3-0b85d4 and the model still answered from
+ # parametric knowledge with executions 0.
+ assert '"function": {"name": "web_search"}' in body
+ assert 'tool_choice = "required"' not in body
+
+
+def test_both_tool_selections_are_tried_before_the_verdict():
+ """One attempt cannot tell a selection bug from a model that will not
+ search.
+
+ `enabled_tools = ["web_search"]` is one name out of ALL_TOOLS, and
+ `routes/inference.py` also reads a request naming only hosted-tool names as
+ a provider-hosted ask. Omitting `enabled_tools` selects every local tool,
+ which is a different path through the same loop. Reporting "the loop
+ offered web_search and never ran it" off the first alone was a guess: no
+ evidence in that run showed the tool had been offered at all.
+ """
+ body = _body()
+ assert '("named", {"enabled_tools": ["web_search"]})' in body
+ assert '("all_local_tools", {})' in body
+ assert '"any_tool_executions"' in body, (
+ "without a count of ANY tool execution, a loop that ran and chose "
+ "something else is indistinguishable from a loop that never ran"
+ )
+
+
+def test_the_second_attempt_is_skipped_once_one_succeeds():
+ """A passing first attempt must not spend a second inference on the same
+ claim; the loop breaks on a positive count."""
+ func = _func("assert_web_search")
+ src = ast.get_source_segment(SRC, func) or ""
+ assert 'if record["executions"]:' in src
+ assert "break" in src
+
+
+def test_the_verdict_counts_web_search_and_not_any_tool():
+ """Mutation found this: summing `any_tool_executions` instead passes on the
+ python tool being run, which is a different assertion in this same payload.
+ The wider count is diagnostic context, never the rule."""
+ func = _func("assert_web_search")
+ verdict = next(
+ node
+ for node in ast.walk(func)
+ if isinstance(node, ast.Assign)
+ and any(
+ isinstance(t, ast.Subscript) and ast.unparse(t) == "detail['executions']"
+ for t in node.targets
+ )
+ )
+ source = ast.unparse(verdict.value)
+ assert "'executions'" in source
+ assert (
+ "any_tool_executions" not in source
+ ), "the verdict counts any tool at all, so the python tool satisfies the web-search claim"
diff --git a/tests/kaggle/test_t4_ci_transport.py b/tests/kaggle/test_t4_ci_transport.py
index 7f824709b31..62c71a9531b 100644
--- a/tests/kaggle/test_t4_ci_transport.py
+++ b/tests/kaggle/test_t4_ci_transport.py
@@ -14,6 +14,7 @@ from __future__ import annotations
import json
import os
+import re
import subprocess
import sys
import tempfile
@@ -23,6 +24,7 @@ import types
from pathlib import Path
import pytest
+import yaml
REPO_ROOT = Path(__file__).resolve().parents[2]
SMOKE_DIR = REPO_ROOT / "tests" / "kaggle" / "t4_smoke"
@@ -34,7 +36,7 @@ sys.path.insert(0, str(CI_DIR))
import build_kernel # noqa: E402
import gate # noqa: E402
import launch # noqa: E402
-from legs import LEGS # noqa: E402
+from legs import KERNELS, LEGS # noqa: E402
# ------------------------------------------------------------------ driver
@@ -57,6 +59,19 @@ class _Stub:
self.gpus = gpus
self.venv_ok = venv_ok
self.papermill: list[dict] = []
+ # Every `pip install --target ...` the driver issued, which is the only
+ # place an overlay's CONTENTS are decided. Recorded as the full command
+ # so a guard can ask what was installed, not merely that something was.
+ self.overlay_installs: list[list[str]] = []
+ # What the resolver is pretended to have found. Deliberately mixed: one
+ # ordinary pure-Python distribution and one native one, so the driver's
+ # deny-list has something real to reject. A stub closure of only safe
+ # packages would let a driver with no deny-list at all pass.
+ self.resolver_closure = [
+ ("transformers", "4.57.6"),
+ ("trl", "0.22.2"),
+ ("torch", "2.99.0"),
+ ]
self.TimeoutExpired = subprocess.TimeoutExpired
self.CalledProcessError = subprocess.CalledProcessError
self.STDOUT = subprocess.STDOUT
@@ -70,6 +85,30 @@ class _Stub:
return types.SimpleNamespace(returncode = 0, stdout = out, stderr = "")
if cmd[0] == "which":
return types.SimpleNamespace(returncode = 0, stdout = "/usr/bin/uv\n", stderr = "")
+ if "--report" in cmd:
+ # `pip install --dry-run --report FILE` writes the resolved closure
+ # to FILE and prints nothing useful, so a stub that only returns a
+ # returncode leaves the driver with an empty manifest -- which it
+ # handles by installing nothing, and the overlay guard would then
+ # pass while proving the overlay never happened.
+ report = Path(cmd[cmd.index("--report") + 1])
+ report.parent.mkdir(parents = True, exist_ok = True)
+ report.write_text(
+ json.dumps(
+ {
+ "install": [
+ {"metadata": {"name": n, "version": v}}
+ for n, v in self.resolver_closure
+ ]
+ }
+ ),
+ encoding = "utf-8",
+ )
+ return types.SimpleNamespace(returncode = 0, stdout = "", stderr = "")
+ if "--target" in cmd:
+ self.overlay_installs.append(list(cmd))
+ Path(cmd[cmd.index("--target") + 1]).mkdir(parents = True, exist_ok = True)
+ return types.SimpleNamespace(returncode = 0, stdout = "", stderr = "")
if "papermill" in cmd:
env = kw.get("env") or {}
self.papermill.append(
@@ -78,6 +117,10 @@ class _Stub:
"cuda": env.get("CUDA_VISIBLE_DEVICES"),
"kernel": cmd[cmd.index("-k") + 1],
"compile_location": env.get("UNSLOTH_COMPILE_LOCATION"),
+ # The whole env, because the caches this file now asserts
+ # on are several variables and a recorder that names them
+ # one at a time goes stale the moment another is added.
+ "env": dict(env),
}
)
Path(cmd[cmd.index("papermill") + 2]).write_text("{}", encoding = "utf-8")
@@ -115,7 +158,15 @@ def _drive(
raised = None
try:
for cell in driver["cells"][:2]:
- source = "".join(cell["source"]).replace("/kaggle/working", str(tmp_path))
+ source = (
+ "".join(cell["source"])
+ # Venvs moved off /kaggle/working (19.5 GB, and the artifact)
+ # onto the ~1 TB overlay when two legs per card made four of
+ # them possible at once. Both roots are rewritten here, or the
+ # stub counts venvs in a directory nothing ever writes to.
+ .replace("/tmp/t4ci_venvs", str(tmp_path / "venvs"))
+ .replace("/kaggle/working", str(tmp_path))
+ )
try:
exec(compile(source, "<driver-cell>", "exec"), namespace)
except SystemExit as exc:
@@ -173,6 +224,7 @@ class _PackedStub(_Stub):
gpus,
durations = None,
hold = 0.05,
+ vram = None,
):
super().__init__(gpus = gpus)
self.durations = durations or {}
@@ -180,35 +232,101 @@ class _PackedStub(_Stub):
self._live_on_card: dict = {}
self._lock = threading.Lock()
self.same_card_overlaps: list = []
+ self.peak_card_gb: dict = {}
+ self.peak_card_legs: dict = {}
+ self.vram = vram or {}
self.max_live_venvs = 0
self.root: Path | None = None
+ self.venv_root: Path | None = None
+ self.venvs_created: list = []
def run(self, cmd, **kw):
cmd = [str(c) for c in cmd]
if len(cmd) > 2 and cmd[1] == "venv":
+ # Recorded at CREATION. Looking for leftover venv_* after the run
+ # cannot tell where they were built: the teardown removes them, so
+ # a kernel building every venv on the 19.5 GB artifact volume ends
+ # just as clean as one building them on the big overlay.
+ self.venvs_created.append(Path(cmd[2]))
Path(cmd[2]).mkdir(parents = True, exist_ok = True)
if "papermill" in cmd:
notebook = Path(cmd[cmd.index("papermill") + 1]).name
card = (kw.get("env") or {}).get("CUDA_VISIBLE_DEVICES")
with self._lock:
- if self._live_on_card.get(card):
- self.same_card_overlaps.append((card, self._live_on_card[card], notebook))
- self._live_on_card[card] = notebook
- if self.root is not None:
- live = len(list(self.root.glob("venv_*")))
- self.max_live_venvs = max(self.max_live_venvs, live)
+ live = self._live_on_card.setdefault(card, set())
+ live.add(notebook)
+ # Two legs on one card is now LEGAL when their measured VRAM
+ # fits, so the overlap itself is no longer the finding. What is
+ # recorded is the peak SUM, which is the thing that has to stay
+ # under budget -- an overlap of two 0.7 GB legs is the feature,
+ # and an overlap involving gptoss at 12.78 GB is the bug.
+ self.peak_card_gb[card] = max(
+ self.peak_card_gb.get(card, 0.0),
+ sum(self.vram.get(n, 1.0) for n in live),
+ )
+ self.peak_card_legs[card] = max(self.peak_card_legs.get(card, 0), len(live))
+ if len(live) > 1:
+ self.same_card_overlaps.append((card, sorted(live)))
+ if self.venv_root is not None:
+ self.max_live_venvs = max(
+ self.max_live_venvs, len(list(self.venv_root.glob("venv_*")))
+ )
time.sleep(self.durations.get(notebook, self.hold))
with self._lock:
- self._live_on_card[card] = None
+ self._live_on_card[card].discard(notebook)
return super().run(cmd, **kw)
+class _HubStub(types.ModuleType):
+ """Records `snapshot_download` calls in order, with a hold.
+
+ The hold is not decoration. The prefetch runs on a thread nobody joins, so
+ an instant stub would let it finish before the first card even starts and
+ every ordering question this file asks would answer itself trivially.
+ """
+
+ def __init__(
+ self,
+ hold = 0.02,
+ fail_for = (),
+ ):
+ super().__init__("huggingface_hub")
+ self.calls: list = []
+ self.hold = hold
+ self.fail_for = set(fail_for)
+ self.hf_home_at_call: list = []
+ # What the hub was asked to FILTER on, per call. Recorded because the
+ # patterns are computed a long way from here and reported in the
+ # summary, and a version that worked out the right glob and then never
+ # passed it would look identical in every artifact.
+ self.patterns_at_call: list = []
+ self._lock = threading.Lock()
+
+ def snapshot_download(
+ self,
+ repo_id = None,
+ **kw,
+ ):
+ with self._lock:
+ self.calls.append(repo_id)
+ self.hf_home_at_call.append(os.environ.get("HF_HOME"))
+ self.patterns_at_call.append(kw.get("allow_patterns"))
+ time.sleep(self.hold)
+ if repo_id in self.fail_for:
+ raise RuntimeError(f"stub refuses {repo_id}")
+
+
def _drive_packed(
tmp_path,
leg_names,
*,
gpus,
durations = None,
+ studio = None,
+ prefetch_repos = (),
+ hub = None,
+ after_gpu_concurrent = False,
+ venv_fallback = False,
):
driver = build_kernel.build_kernel(
SMOKE_DIR,
@@ -218,43 +336,184 @@ def _drive_packed(
extra_args = (),
per_run_timeout = 60,
skip_reference = True,
+ studio = studio,
+ prefetch_repos = prefetch_repos,
+ after_gpu_concurrent = after_gpu_concurrent,
+ )
+ stub = _PackedStub(
+ gpus = gpus,
+ durations = durations,
+ vram = {f"t4_{n}.ipynb": LEGS[n].vram_gb for n in leg_names},
)
- stub = _PackedStub(gpus = gpus, durations = durations)
stub.root = tmp_path
+ # On the fallback path the venvs land in WORK itself, so that is where the
+ # stub has to count them.
+ stub.venv_root = tmp_path if venv_fallback else tmp_path / "venvs"
+ hub = hub if hub is not None else _HubStub()
saved = sys.modules["subprocess"]
+ saved_hub = sys.modules.get("huggingface_hub")
sys.modules["subprocess"] = stub
+ sys.modules["huggingface_hub"] = hub
namespace: dict = {}
raised = None
try:
for cell in driver["cells"][:2]:
- source = "".join(cell["source"]).replace("/kaggle/working", str(tmp_path))
+ source = (
+ "".join(cell["source"])
+ # Venvs moved off /kaggle/working (19.5 GB, and the artifact)
+ # onto the ~1 TB overlay when two legs per card made four of
+ # them possible at once. Both roots are rewritten here, or the
+ # stub counts venvs in a directory nothing ever writes to.
+ # venv_fallback points the preferred root at a path whose
+ # parent is a regular file, so `mkdir` raises OSError and the
+ # kernel takes its own fallback branch. Rewriting it straight
+ # to WORK would test an assignment; this tests the branch.
+ .replace(
+ "/tmp/t4ci_venvs",
+ str(tmp_path / "blocked" / "t4ci_venvs")
+ if venv_fallback
+ else str(tmp_path / "venvs"),
+ )
+ .replace("/kaggle/working", str(tmp_path))
+ )
try:
exec(compile(source, "<driver-cell>", "exec"), namespace)
except SystemExit as exc:
raised = exc
break
+ # JOIN the lane before handing back. The kernel deliberately does not
+ # (it is a daemon thread, so a slow download cannot hold the session
+ # open), but a test that merely SAMPLES it leaks: the lane resolves
+ # `huggingface_hub` out of sys.modules at call time, so one still
+ # running after its test returns records into the NEXT test's stub.
+ # That is not hypothetical -- it is how this harness first went order
+ # dependent, passing alone and failing inside the suite.
+ lane = namespace.get("prefetch_thread")
+ if lane is not None:
+ lane.join(30.0)
+ assert not lane.is_alive(), "the prefetch lane outlived its test"
finally:
sys.modules["subprocess"] = saved
- return {"stood_down": raised, "stub": stub, "results": namespace.get("results") or {}}
+ if saved_hub is None:
+ sys.modules.pop("huggingface_hub", None)
+ else:
+ sys.modules["huggingface_hub"] = saved_hub
+ return {
+ "stood_down": raised,
+ "stub": stub,
+ "hub": hub,
+ "results": namespace.get("results") or {},
+ "card_load": namespace.get("card_load") or {},
+ "card_count": namespace.get("card_count") or {},
+ }
-ALL_FOUR = ["gptoss", "frontier", "canary", "control"]
+# Derived from KERNELS, not a second copy of it. As a literal this silently
+# went on describing the OLD longest-first order after legs.py moved to the
+# second-wave one, so every test driving it was exercising an order the kernel
+# no longer builds -- including the test that exists to assert the order.
+# Derived from KERNELS, so it follows the registry rather than restating it.
+# Renamed from ALL_LEGS when the Default leg made the kernel five: a name that
+# counts is a name that goes stale silently, and two assertions below had
+# already hardcoded the 4 to match it.
+# Every wired leg, INCLUDING the all-card one, because the scheduling rules
+# below have to hold for the set that actually ships.
+#
+# This was narrowed to exclude all-card legs for one commit, on the reasoning
+# that a leg outside the card queue cannot answer a question about who shares a
+# card. The reasoning was half right and the narrowing was hiding a bug: the
+# all-card leg was charging a card_count slot, so it really did cost the queue
+# a slot on a card, and the co-tenancy rule really was failing. The count is
+# global now and the rule holds with the leg present, so the wider set is back.
+ALL_LEGS = list(KERNELS[0])
+
+
+def test_losing_tmp_drops_the_kernel_back_to_one_leg_per_card(tmp_path):
+ """The venv fallback described an intention nothing implemented.
+
+ Venvs moved to /tmp because co-scheduling made four torch-bearing venvs
+ possible at once and four do not fit in the 19.5 GB /kaggle/working. The
+ fallback for a box with no writable /tmp says it "keeps a one-leg-per-card
+ run working" -- but MAX_LEGS_PER_CARD was a constant, so the fallback put
+ the venvs back on the small partition and went right on building two per
+ card. It would have surfaced as an install dying halfway through, which
+ reads like anything except a full disk.
+
+ The fallback BRANCH is exercised, not simulated: the preferred root is
+ pointed at a path whose parent is a regular file, so mkdir raises exactly
+ as it would there.
+ """
+ (tmp_path / "blocked").write_text("not a directory")
+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2, venv_fallback = True)
+ assert driven["stood_down"] is None
+ stub = driven["stub"]
+ assert stub.venv_root is not None
+ for card, count in stub.peak_card_legs.items():
+ assert count <= 1, (
+ f"card {card} ran {count} legs at once with the venvs back on "
+ f"/kaggle/working: {stub.same_card_overlaps}"
+ )
+ assert stub.max_live_venvs <= 2, stub.max_live_venvs
+ assert len(stub.papermill) == len(ALL_LEGS), stub.papermill
-def test_four_legs_on_two_cards_never_put_two_legs_on_one_card_at_once(tmp_path):
- """The property that makes packing safe at all.
+def test_a_seeds_seat_is_taken_before_any_worker_can_look_at_the_card(tmp_path):
+ """The race that put 13.48 GB on a 13.0 GB card, on real hardware.
- Four payloads across two T4s is only sound because a card takes its next
- leg when the previous one has EXITED. If they overlapped, each child would
- still pass its own `device_count() == 1` assertion and then fight for 15GB,
- which is exactly the failure the shortfall guard was added to prevent and
- exactly the one it cannot see from where it stands.
+ `test_no_card_is_ever_asked_to_hold_more_than_it_has` asserts the same
+ budget and passed throughout, because with default stub durations every
+ leg finishes before the 5s start stagger elapses and no overlap is ever
+ recorded. Run 32667451396 was not so lucky: gpu1's seed sat unreserved for
+ those 5s, a free worker saw an empty card and put gptoss on it, and when
+ the seed worker finally woke, `_admit` correctly refused and the caller
+ threw the answer away. control and gptoss then shared one card for 691s.
+
+ So the durations here are chosen to hold the window open rather than to be
+ fast: gptoss must outlive the stagger, or the second leg lands after it has
+ already finished and the test goes green on a schedule that never happened.
+ """
+ driven = _drive_packed(
+ tmp_path,
+ ALL_LEGS,
+ gpus = 2,
+ durations = {
+ "t4_canary.ipynb": 2.0,
+ "t4_control.ipynb": 2.0,
+ "t4_frontier.ipynb": 0.2,
+ "t4_gptoss.ipynb": 7.0,
+ },
+ )
+ stub = driven["stub"]
+ for card, peak in stub.peak_card_gb.items():
+ assert peak <= 13.0, f"card {card} peaked at {peak} GB. Overlaps: {stub.same_card_overlaps}"
+ # gptoss is 12.78 of a 13.0 budget, so it can only ever run alone. Asserted
+ # on the overlap record as well as the sum: a VRAM table that silently
+ # under-priced it would satisfy the sum check while the card burned.
+ for card, live in stub.same_card_overlaps:
+ assert "t4_gptoss.ipynb" not in live, (card, live)
+
+
+def test_no_card_is_ever_asked_to_hold_more_than_it_has(tmp_path):
+ """Was "never two legs on one card"; is now "never over the VRAM budget".
+
+ Two legs on a card is the FEATURE, not the bug: measured on run
+ 32611343797 the three Qwen legs peak at 0.70 GB each on a 14.56 GB card,
+ so one leg per card left it 95% empty. What must still never happen is the
+ thing that produced the OOM this file's shortfall guard was written for --
+ payloads whose summed appetite exceeds the card. gptoss peaks at 12.78 GB,
+ so it is excluded by the arithmetic rather than by a special case.
+
+ Asserted on the summed GB and not on the overlap, because after this change
+ an overlap is exactly what success looks like.
"""
- driven = _drive_packed(tmp_path, ALL_FOUR, gpus = 2)
+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2)
assert driven["stood_down"] is None
stub = driven["stub"]
- assert stub.same_card_overlaps == [], stub.same_card_overlaps
- assert len(stub.papermill) == 4, stub.papermill
+ for card, peak in stub.peak_card_gb.items():
+ assert peak <= 13.0, f"card {card} peaked at {peak} GB: {stub.same_card_overlaps}"
+ for card, count in stub.peak_card_legs.items():
+ assert count <= 2, f"card {card} held {count} legs at once"
+ assert len(stub.papermill) == len(ALL_LEGS), stub.papermill
# Both cards are used, and every leg ran. The SPLIT is deliberately not
# asserted: how many legs each card ends up with is a function of how long
# the legs take relative to the 5s venv stagger, not something the
@@ -263,24 +522,59 @@ def test_four_legs_on_two_cards_never_put_two_legs_on_one_card_at_once(tmp_path)
# and the split is 2/2; under the sub-second stubs here the first card
# legitimately drains most of the queue before the second clears its
# stagger. Pinning 2/2 would be pinning the stub's timing.
- assert set(p["cuda"] for p in stub.papermill) == {"0", "1"}, stub.papermill
-
-
-def test_the_longest_leg_starts_first_so_the_schedule_can_balance_around_it(tmp_path):
- """Start order is longest-first, and it is load bearing rather than tidy.
-
- Measured on run 32607621452: gptoss 384.1s, frontier 312.2s, canary 265.3s,
- control 262.2s. Longest-first packs those as 646.3s, which is the optimal
- split of the four; `sorted(PAYLOADS)` would start gptoss LAST, and a greedy
- scheduler cannot balance around the leg that sets the makespan if it picks
- it up at the end.
+ # Both cards pinned, across the payloads that HAVE a card index. An
+ # all-card leg is unpinned by design -- that is the whole reason it exists,
+ # since unsloth binds its DEVICE_COUNT > 1 helpers on visibility -- so it
+ # inherits the ambient CUDA_VISIBLE_DEVICES exactly as the Studio lanes do,
+ # and asserting it against a card index tests the box this ran on.
+ pinned = [
+ p
+ for p in stub.papermill
+ if p["notebook"] not in {f"t4_{LEGS[n].name}.ipynb" for n in ALL_LEGS if LEGS[n].all_cards}
+ ]
+ assert set(p["cuda"] for p in pinned) == {"0", "1"}, stub.papermill
+ # ...and if any all-card leg is WIRED, it really did run and really did run
+ # unpinned. Conditional because the wired set has none today: multi_gpu was
+ # measured and held out. Asserting one unconditionally would go red on a
+ # scheduling change that is nothing to do with this rule.
+ unpinned = [p for p in stub.papermill if p not in pinned]
+ if any(LEGS[n].all_cards for n in ALL_LEGS):
+ assert unpinned, "an all-card leg is wired and no unpinned payload ran"
+ else:
+ assert not unpinned, unpinned
+
+
+def test_gptoss_starts_in_the_second_wave_so_the_prefetch_has_a_window(tmp_path):
+ """Was longest-first; is now second-wave, and the change is the point.
+
+ A prefetch only pays for what it finishes BEFORE the leg that wants the
+ model starts, and gptoss is the only leg with a ~12 GB download. Starting
+ it at t=0 leaves no window in front of it, which is why prefetching without
+ this reorder measures WORSE than doing neither (603.1s against 563.1s).
+
+ Third is first pick of the second wave: with two cards, positions 0 and 1
+ are seeded and 2 is the first to be taken off the pending queue, so gptoss
+ starts at ~190-220s under the measured durations. That is lead time the
+ prefetch spends, and a small leg is still running beside it.
+
+ Asserted on POSITION in the order rather than on a start timestamp: the
+ timestamp is a function of the stub's durations, and pinning it would pin
+ the stub. legs.KERNELS carries the full table.
"""
- driven = _drive_packed(tmp_path, ALL_FOUR, gpus = 2)
+ order = list(KERNELS[0])
+ assert order.index("gptoss") == 2, (
+ f"gptoss is at position {order.index('gptoss')} of {order}; first means "
+ "the prefetch has no window and second-wave is what buys the saving"
+ )
+ # ...and it must not be LAST either, which is the other intuitive answer.
+ # gptoss is the longest leg, so ending on it idles the other card for its
+ # whole ~284s: simulated at 651.1s worst case against 528.1s here.
+ assert order[-1] != "gptoss", order
+
+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2)
started = [p["notebook"] for p in driven["stub"].papermill]
- assert started[0] == "t4_gptoss.ipynb", started
- assert started != sorted(
- started
- ), "payloads are running in alphabetical order, so the longest leg is last"
+ assert started[0] != "t4_gptoss.ipynb", started
+ assert started != sorted(started), "payloads are running in alphabetical order"
def test_each_leg_keeps_its_own_venv_compile_cache_and_ipykernel(tmp_path):
@@ -292,7 +586,7 @@ def test_each_leg_keeps_its_own_venv_compile_cache_and_ipykernel(tmp_path):
an index reused across a wave would silently merge two legs' trees and the
last writer would win.
"""
- driven = _drive_packed(tmp_path, ALL_FOUR, gpus = 2)
+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2)
calls = driven["stub"].papermill
for field in ("kernel", "compile_location", "notebook"):
values = [c[field] for c in calls]
@@ -300,19 +594,220 @@ def test_each_leg_keeps_its_own_venv_compile_cache_and_ipykernel(tmp_path):
def test_a_finished_leg_gives_its_virtualenv_back(tmp_path):
- """Otherwise four torch trees sit on /kaggle/working at once.
+ """Each venv carries its own torch and its own NVIDIA runtime.
The tail cell prunes `venv_*`, but only after every payload has finished,
- which was sufficient when a kernel held one payload per card. Packed, the
- peak is what matters, and it has to stay at one venv per CARD rather than
- one per LEG.
+ so the PEAK is what matters and freeing at the end of each leg is what
+ bounds it. The bound is one venv per concurrent LEG, which co-scheduling
+ raised from 2 to 4.
+
+ That raise is precisely why the venvs no longer live on `/kaggle/working`.
+ That path is 19.5 GB and is also what Kaggle ships home; four torch trees
+ do not fit in it, and the failure would arrive as an install dying midway
+ for reasons that look nothing like a full disk. They go on the ~1 TB
+ overlay instead, and only the evidence stays where Kaggle collects it.
"""
- driven = _drive_packed(tmp_path, ALL_FOUR, gpus = 2)
+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2)
stub = driven["stub"]
+ ceiling = 2 * 2 # cards x MAX_LEGS_PER_CARD
assert (
- stub.max_live_venvs <= 2
- ), f"{stub.max_live_venvs} virtualenvs were alive at once on a 2-card kernel"
- assert list(tmp_path.glob("venv_*")) == [], "a payload left its virtualenv behind"
+ stub.max_live_venvs <= ceiling
+ ), f"{stub.max_live_venvs} virtualenvs were alive at once, ceiling {ceiling}"
+ assert list((tmp_path / "venvs").glob("venv_*")) == [], "a payload left its virtualenv behind"
+ # ...and none of them was ever created on the artifact volume.
+ assert stub.venvs_created, "no virtualenv was built at all"
+ for created in stub.venvs_created:
+ assert (tmp_path / "venvs") in created.parents, (
+ f"a virtualenv was built at {created}, on /kaggle/working -- which "
+ "is 19.5 GB and is also the artifact Kaggle ships home"
+ )
+
+
+# --------------------------------------------------- Studio in the same kernel
+
+STUDIO = {
+ "unsloth_ref": "main",
+ "repo_url": "https://github.com/unslothai/unsloth",
+ "payload_args": "--max-steps 8",
+}
+STUDIO_INSTALL = build_kernel.STUDIO_INSTALL_NOTEBOOK
+STUDIO_TEST = build_kernel.STUDIO_TEST_NOTEBOOK
+# A value no card index can be confused with, so "the driver left this alone"
+# and "the driver pinned a card" are distinguishable. An unpinned lane inherits
+# whatever the ambient environment has; a pinned one is overwritten with a
+# single index.
+AMBIENT_CUDA = "0,1"
+
+
+def _drive_with_studio(
+ tmp_path,
+ monkeypatch,
+ leg_names,
+ *,
+ gpus = 2,
+ durations = None,
+):
+ monkeypatch.setenv("CUDA_VISIBLE_DEVICES", AMBIENT_CUDA)
+ return _drive_packed(tmp_path, leg_names, gpus = gpus, durations = durations, studio = STUDIO)
+
+
+def test_the_studio_install_never_takes_a_card_and_the_legs_never_wait_for_it(
+ tmp_path, monkeypatch
+):
+ """The whole point of carrying Studio here: its install is free time.
+
+ Checkout, `install.sh --local`, the frontend build and the Playwright
+ browser are network and CPU and touch no GPU, so they run beside the
+ training legs rather than after them. If this lane ever queued for a card
+ it would displace a leg and the merge would cost more than it saves.
+ """
+ driven = _drive_with_studio(
+ tmp_path,
+ monkeypatch,
+ ALL_LEGS,
+ durations = {f"t4_{LEGS[n].name}.ipynb": 0.30 for n in ALL_LEGS},
+ )
+ assert driven["stood_down"] is None
+ calls = {c["notebook"]: c for c in driven["stub"].papermill}
+ assert STUDIO_INSTALL in calls, sorted(calls)
+
+ # Unpinned. install.sh --local resolves torch, and an installer that cannot
+ # see a device resolves a CPU-only one -- which is the exact regression
+ # Studio's verify cell exists to catch, so hiding the cards here would
+ # manufacture it.
+ assert calls[STUDIO_INSTALL]["cuda"] == AMBIENT_CUDA, calls[STUDIO_INSTALL]
+ # ...while every leg is still pinned to exactly one card. The SPLIT is not
+ # asserted, for the reason given in
+ # test_four_legs_on_two_cards_never_put_two_legs_on_one_card_at_once: how
+ # many legs each card takes depends on leg duration against the 5s venv
+ # stagger, and under sub-second stubs the first card legitimately drains
+ # most of the queue.
+ # "every leg" here means every leg that HAS a card. An all-card leg is
+ # unpinned by design and inherits the ambient CUDA_VISIBLE_DEVICES, like the
+ # Studio lanes above; asserting it against a card index tests the box the
+ # suite ran on rather than the scheduler.
+ unpinned = {f"t4_{LEGS[n].name}.ipynb" for n in ALL_LEGS if LEGS[n].all_cards}
+ leg_cards = [c["cuda"] for n, c in calls.items() if n.startswith("t4_") and n not in unpinned]
+ assert len(leg_cards) == len(ALL_LEGS) - len(unpinned), calls
+ assert set(leg_cards) == {"0", "1"}, leg_cards
+ # Every all-card leg really did run, and really did run unpinned. Without
+ # this the exclusion above could quietly cover a leg that never started.
+ for notebook in unpinned:
+ assert notebook in calls, sorted(calls)
+ assert calls[notebook]["cuda"] == AMBIENT_CUDA, calls[notebook]
+ # Two legs on a card is legal now (see the VRAM budget); what must hold is
+ # that the summed appetite never exceeds what the card has.
+ for card, peak in driven["stub"].peak_card_gb.items():
+ assert peak <= 13.0, (card, peak, driven["stub"].same_card_overlaps)
+
+
+def test_the_studio_assertions_wait_for_both_cards_rather_than_borrowing_one(tmp_path, monkeypatch):
+ """Studio keeps both T4s visible, and that is deliberate upstream.
+
+ Its own driver says so: "Studio's own device selection is part of what is
+ under test; masking one would test a machine nobody has." So the GPU half
+ runs once the leg queue has drained, unpinned, rather than being handed a
+ single card out of the queue.
+ """
+ driven = _drive_with_studio(tmp_path, monkeypatch, ALL_LEGS)
+ calls = [c["notebook"] for c in driven["stub"].papermill]
+ assert STUDIO_TEST in calls, calls
+ # Last, after every leg.
+ assert calls[-1] == STUDIO_TEST, calls
+ by_name = {c["notebook"]: c for c in driven["stub"].papermill}
+ assert by_name[STUDIO_TEST]["cuda"] == AMBIENT_CUDA, by_name[STUDIO_TEST]
+
+
+def test_a_failed_studio_install_skips_its_assertions_with_the_reason(tmp_path, monkeypatch):
+ """Otherwise the missing venv is reported as a Studio regression.
+
+ The install half is what puts the interpreter, the frontend and the
+ llama.cpp on disk. Running the assertions against a half-built tree fails
+ on `no interpreter at ...`, which reads like the code under test broke
+ rather than like the install did.
+ """
+ monkeypatch.setenv("CUDA_VISIBLE_DEVICES", AMBIENT_CUDA)
+
+ class _InstallFails(_PackedStub):
+ def run(self, cmd, **kw):
+ cmd = [str(c) for c in cmd]
+ if "papermill" in cmd and STUDIO_INSTALL in " ".join(cmd):
+ self.papermill.append(
+ {
+ "notebook": STUDIO_INSTALL,
+ "cuda": None,
+ "kernel": None,
+ "compile_location": None,
+ }
+ )
+ Path(cmd[cmd.index("papermill") + 2]).write_text("{}", encoding = "utf-8")
+ return types.SimpleNamespace(returncode = 1, stdout = "", stderr = "")
+ return super().run(cmd, **kw)
+
+ driver = build_kernel.build_kernel(
+ SMOKE_DIR,
+ ALL_LEGS,
+ unsloth_ref = "main",
+ zoo_ref = "main",
+ extra_args = (),
+ per_run_timeout = 60,
+ skip_reference = True,
+ studio = STUDIO,
+ )
+ stub = _InstallFails(gpus = 2)
+ stub.root = tmp_path
+ stub.venv_root = tmp_path / "venvs"
+ saved = sys.modules["subprocess"]
+ sys.modules["subprocess"] = stub
+ namespace: dict = {}
+ try:
+ for cell in driver["cells"][:2]:
+ source = (
+ "".join(cell["source"])
+ # Venvs moved off /kaggle/working (19.5 GB, and the artifact)
+ # onto the ~1 TB overlay when two legs per card made four of
+ # them possible at once. Both roots are rewritten here, or the
+ # stub counts venvs in a directory nothing ever writes to.
+ .replace("/tmp/t4ci_venvs", str(tmp_path / "venvs"))
+ .replace("/kaggle/working", str(tmp_path))
+ )
+ exec(compile(source, "<driver-cell>", "exec"), namespace)
+ finally:
+ sys.modules["subprocess"] = saved
+
+ ran = [c["notebook"] for c in stub.papermill]
+ assert STUDIO_TEST not in ran, ran
+ # Every leg still ran: a broken Studio install must not take the notebook
+ # signal down with it.
+ assert sorted(n for n in ran if n.startswith("t4_")) == sorted(
+ f"t4_{LEGS[leg].name}.ipynb" for leg in ALL_LEGS
+ )
+ recorded = (namespace.get("results") or {}).get(STUDIO_TEST)
+ assert recorded is not None, "the skip was not recorded at all"
+ assert recorded["returncode"] is None
+ assert "install lane did not succeed" in recorded["error"]
+
+
+def test_studio_is_not_in_the_card_queue(tmp_path, monkeypatch):
+ """ORDER is the legs. Either Studio half in it would be handed a card."""
+ driver = build_kernel.build_kernel(
+ SMOKE_DIR,
+ ALL_LEGS,
+ unsloth_ref = "main",
+ zoo_ref = "main",
+ extra_args = (),
+ per_run_timeout = 60,
+ skip_reference = True,
+ studio = STUDIO,
+ )
+ setup = "".join(driver["cells"][0]["source"])
+ order = next(l for l in setup.splitlines() if l.startswith("ORDER = "))
+ assert STUDIO_INSTALL not in order, order
+ assert STUDIO_TEST not in order, order
+ assert order.count("t4_") == len(ALL_LEGS), order
+ # ...but both are carried, or the kernel would have nothing to run.
+ payloads = set(driver["metadata"]["kaggle_t4_ci"]["payloads"])
+ assert {STUDIO_INSTALL, STUDIO_TEST} <= payloads, sorted(payloads)
def test_a_one_card_allocation_still_stands_a_packed_kernel_down(tmp_path):
@@ -325,7 +820,7 @@ def test_a_one_card_allocation_still_stands_a_packed_kernel_down(tmp_path):
infrastructure, because one card silently serialises the whole kernel and
doubles its wall clock while looking like a slow but healthy run.
"""
- driven = _drive_packed(tmp_path, ALL_FOUR, gpus = 1)
+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 1)
assert driven["stood_down"] is not None, "a 1-GPU allocation ran the packed kernel anyway"
assert driven["stub"].papermill == []
@@ -1853,3 +2348,961 @@ def test_main_bounds_the_whole_evidence_phase_it_is_budgeted_for(monkeypatch, tm
assert seen[0] is not None, "main() never handed the collection a deadline"
assert seen[0] == seen[1], "the two kernels must share ONE budget, not get one each"
assert seen[0] - started <= launch.EVIDENCE_BUDGET_SEC
+
+
+# --------------------------------------------------------- the merged kernel's
+# --------------------------------------------------------- two reporters
+
+NOTEBOOK_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "kaggle-t4-notebook-ci.yml"
+
+
+def test_the_merged_kernel_runs_both_reporters():
+ """One kernel, two experiments, so two report steps over one evidence dir.
+
+ Dropping the Studio one is the failure this guard exists for: the kernel
+ would still install Studio, still drive the UI on a T4, and the job would
+ still go green with nothing said about it. The T4 reporter FILTERS the
+ studio-gpu label out, so its section would look complete while the payload
+ that half the wall clock went on is unreported.
+ """
+ source = NOTEBOOK_WORKFLOW.read_text(encoding = "utf-8")
+ assert ".github/scripts/kaggle_t4_ci/report.py" in source
+ assert ".github/scripts/kaggle_studio_ci/report.py" in source
+ assert ".github/scripts/kaggle_studio_ci/collect_evidence.py" in source
+
+
+def test_the_shared_wheels_are_the_specs_every_leg_holds_in_common():
+ """Built once from the very SHAs the legs name -- main AND the ref tested.
+
+ Every leg installs unsloth_zoo and unsloth from the same two pinned SHAs,
+ and pip does not cache a VCS build, so run 32679427416 cloned and built
+ both FOUR times: install was 149-191s per leg, the largest single phase of
+ each and 41% of gpt-oss.
+
+ The list is DERIVED from the legs' own groups, never declared again, and
+ that is what keeps it honest. A hand-written copy of the two SHAs would
+ drift silently, because a wheel built from the wrong ref installs perfectly
+ and fails nothing at all.
+
+ The intersection is the safety property, not an optimisation: a spec only
+ one leg carries is part of what that leg tests, and sharing it would make
+ the legs agree about the thing they exist to disagree about.
+ """
+ common = build_kernel._shared_vcs_specs(
+ {
+ "a": [["unsloth_zoo @ git+u@S1"], ["transformers==5.5.0"]],
+ "b": [["unsloth_zoo @ git+u@S1"], ["--upgrade", "transformers"]],
+ }
+ )
+ assert common == ("unsloth_zoo @ git+u@S1",), common
+
+ # Different refs for the same package share NOTHING. Returning either one
+ # would hand a leg a wheel built from the other leg's commit.
+ assert (
+ build_kernel._shared_vcs_specs(
+ {
+ "a": [["unsloth @ git+u@S1"]],
+ "b": [["unsloth @ git+u@S2"]],
+ }
+ )
+ == ()
+ )
+
+ # A spec only one leg carries is never shared.
+ assert build_kernel._shared_vcs_specs(
+ {
+ "a": [["x @ git+u@S1"], ["y @ git+u@S9"]],
+ "b": [["x @ git+u@S1"]],
+ }
+ ) == ("x @ git+u@S1",)
+
+ # And on the real legs: BOTH packages, at the refs asked for.
+ driver = build_kernel.build_kernel(
+ SMOKE_DIR,
+ ALL_LEGS,
+ unsloth_ref = "PRSHA",
+ zoo_ref = "MAINSHA",
+ extra_args = (),
+ per_run_timeout = 60,
+ skip_reference = True,
+ shared_wheels = True,
+ )
+ src = "".join("".join(c["source"]) for c in driver["cells"])
+ specs = re.search(r"SHARED_WHEEL_SPECS = (.+)", src).group(1)
+ assert "unsloth @ git+" in specs and "unsloth_zoo @ git+" in specs, specs
+ assert "PRSHA" in specs and "MAINSHA" in specs, specs
+ # Built before any leg can start. A background build loses the race it
+ # exists to win: the legs are admitted and start installing at t=0.
+ assert src.index('"pip", "wheel"') < src.index(
+ "threads = []"
+ ), "the wheels are built after the leg workers start, so no leg can use them"
+
+
+def test_every_leg_gets_its_own_torch_and_triton_cache(tmp_path):
+ """A separate venv never protected these, and nobody noticed.
+
+ UNSLOTH_COMPILE_LOCATION was set per leg and assumed to be the whole story.
+ torch and triton key their caches off $TMPDIR rather than off the
+ interpreter: on torch 2.9.1 an unset TORCHINDUCTOR_CACHE_DIR resolves to
+ `tempfile.gettempdir()/torchinductor_$USER`
+ (torch/_inductor/runtime/cache_dir_utils.py:22) and the triton cache lands
+ under that same directory. So four legs whose entire purpose is to install
+ DIFFERENT transformers/TRL/peft versions and compile the same modules were
+ sharing one /tmp/torchinductor_root.
+
+ Asserted as DISTINCT per leg rather than merely present -- one directory
+ named once and handed to everybody would satisfy "is set" and reproduce the
+ bug exactly.
+ """
+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2)
+ seen: dict[str, set] = {}
+ for call in driven["stub"].papermill:
+ env = call.get("env") or {}
+ for key in (
+ "TORCHINDUCTOR_CACHE_DIR",
+ "TRITON_CACHE_DIR",
+ "TMPDIR",
+ "UNSLOTH_COMPILE_LOCATION",
+ ):
+ assert env.get(key), f"{call['notebook']} has no {key}"
+ seen.setdefault(key, set()).add(env[key])
+ for key, values in seen.items():
+ assert len(values) == len(
+ driven["stub"].papermill
+ ), f"{key} is shared between legs: {sorted(values)}"
+
+
+def test_two_dispatches_can_hold_the_two_kaggle_slots_at_once():
+ """One concurrency group cannot express "at most two", and one was used.
+
+ A Kaggle account allows 2 concurrent GPU sessions and this job takes one,
+ so the cap is 2 -- but GitHub concurrency is 1 per group. The single group
+ did not merely serialise: only ONE run may be PENDING in a group, so a
+ second queued run CANCELS the first instead of queueing behind it. That
+ killed run 32674255736 and made an A/B impossible to run at all, which is
+ how two "different" configurations came to be compared against each other
+ while executing the same schedule.
+
+ Bounded by construction is the property worth guarding: the input offers
+ exactly two slots, so the account can never be asked for a third session.
+ """
+ source = NOTEBOOK_WORKFLOW.read_text(encoding = "utf-8")
+ workflow = yaml.safe_load(source)
+ slot = workflow[True]["workflow_dispatch"]["inputs"]["slot"]
+ assert slot["type"] == "choice", slot
+ assert slot["options"] == ["1", "2"], (
+ f"the slot input offers {slot['options']}, so the account could be "
+ f"asked for more than its 2 concurrent sessions"
+ )
+ assert slot.get("default") == "1", slot
+
+ # BOTH levels, because the discard rule applies to both and fixing only the
+ # job left the whole thing broken in exactly the same way: the two runs
+ # still shared one workflow-level group, so the second dispatch discarded
+ # the first while it was pending and never reached the job group at all.
+ for scope, block in (
+ ("workflow", workflow["concurrency"]),
+ ("job", workflow["jobs"]["t4-smoke"]["concurrency"]),
+ ):
+ group = block["group"]
+ assert "inputs.slot" in group, f"{scope}: {group}"
+ # A non-dispatch event has no input and must land in a single shared
+ # slot, or every push would get a session of its own.
+ assert "'1'" in group, f"{scope}: {group}"
+ assert block["cancel-in-progress"] is False, scope
+
+
+def test_the_shared_wheel_build_is_opt_in():
+ """Measured once, attributable to nothing, so it ships behind a flag.
+
+ On run 32689629906 the wheels helped the leg that runs ALONE (gpt-oss
+ install 191.2s -> 152.6s) and cost the three that run CONCURRENTLY
+ (149-163s -> 319-334s). That run also changed the torch/triton cache
+ layout, so neither effect can be attributed to either change. A default-on
+ optimisation resting on that would be a guess wearing a measurement's
+ clothes.
+ """
+ source = NOTEBOOK_WORKFLOW.read_text(encoding = "utf-8")
+ workflow = yaml.safe_load(source)
+ inputs = workflow[True]["workflow_dispatch"]["inputs"]
+ assert inputs["shared_wheels"].get("default") is False, inputs["shared_wheels"]
+ build = source.split("build_kernel.py")[1].split("- name:")[0]
+ assert "$SHARED_WHEELS" in build, build
+ assert "'--shared-wheels'" in source
+
+ # Off means no wheel build in the kernel at all, not merely an unused one.
+ off = build_kernel.build_kernel(
+ SMOKE_DIR,
+ ALL_LEGS,
+ unsloth_ref = "R",
+ zoo_ref = "R",
+ extra_args = (),
+ per_run_timeout = 60,
+ skip_reference = True,
+ )
+ src = "".join("".join(c["source"]) for c in off["cells"])
+ assert "SHARED_WHEEL_SPECS = ()" in src, (
+ "wheels are off but the kernel still carries specs, so it would spend "
+ "the build time and change nothing"
+ )
+
+
+def test_the_workflow_can_actually_reach_studio_concurrent():
+ """A CLI flag nothing passes is dead code that reads as a feature.
+
+ This is not hypothetical. Run 32674263571 was dispatched as the VARIANT of
+ an A/B on exactly this behaviour. `--studio-concurrent` existed in
+ build_kernel's argument parser and was threaded all the way to
+ AFTER_GPU_CONCURRENT, the unit tests for it passed, and the workflow never
+ passed the flag -- so the kernel built with it False, the "variant" ran the
+ control's schedule, and the comparison was a configuration against itself.
+ Nothing was red. `AFTER_GPU_SHARED` was simply absent from kernel.log, and
+ absence is not something a green tick reports.
+
+ So the chain is asserted end to end: the input exists, something converts
+ it into the flag, and the flag reaches the build command.
+ """
+ source = NOTEBOOK_WORKFLOW.read_text(encoding = "utf-8")
+ workflow = yaml.safe_load(source)
+ inputs = workflow[True]["workflow_dispatch"]["inputs"]
+ assert "studio_concurrent" in inputs, sorted(inputs)
+ # ON by default now, and the reason is a measurement: Studio's GPU half is
+ # ~1100s and it used to start only once every leg had freed both cards, on
+ # top of a ~1200s leg phase, so sharing takes that block off the critical
+ # path. What it costs -- Studio seeing one T4 rather than two -- is not
+ # silent: the payload sizes tensor_split to the visible cards and records
+ # `tensor_split_over_two_cards: false` with a note, which is asserted by
+ # test_studio_server_flags.py.
+ assert inputs["studio_concurrent"].get("default") is True, inputs["studio_concurrent"]
+ # And it must still be switchable OFF, or the both-cards run is unreachable.
+ assert 'inputs.studio_concurrent }}" = "false"' in source, (
+ "nothing reads the input as a way to turn sharing off, so a dispatch "
+ "asking for the two-card coverage would silently get the shared shape"
+ )
+
+ build = source.split("build_kernel.py")[1].split("- name:")[0]
+ assert "$STUDIO_CONCURRENT" in build, (
+ "the build command does not interpolate STUDIO_CONCURRENT, so the "
+ "input cannot reach the kernel no matter what it is set to"
+ )
+ assert "inputs.studio_concurrent" in source
+ assert "'--studio-concurrent'" in source or '"--studio-concurrent"' in source
+
+ # And the flag the workflow spells must be one the CLI accepts. A rename on
+ # either side would otherwise land as an unrecognised argument at build
+ # time, or worse, be silently ignored.
+ cli = (CI_DIR / "build_kernel.py").read_text(encoding = "utf-8")
+ assert '"--studio-concurrent"' in cli, "build_kernel.py does not define the flag"
+
+
+def test_the_t4_reporter_is_told_the_leg_count_not_the_payload_count():
+ """`payloads` counts Studio; `legs` does not, and this reporter drops it.
+
+ Handing it `payloads` makes a complete four-leg result read as short by one
+ forever: it filters the studio-gpu report out and then compares what is
+ left against a number that included it.
+ """
+ source = NOTEBOOK_WORKFLOW.read_text(encoding = "utf-8")
+ reporter = source.split(".github/scripts/kaggle_t4_ci/report.py")[1].split("- name:")[0]
+ assert "steps.build.outputs.legs" in reporter
+ assert "steps.build.outputs.payloads" not in reporter
+
+
+ ("label", "reporter", "expect_red"),
+ [
+ ("control", "kaggle_t4_ci", True),
+ ("control", "kaggle_studio_ci", False),
+ ("studio-gpu", "kaggle_t4_ci", False),
+ ("studio-gpu", "kaggle_studio_ci", True),
+ ],
+)
+def test_a_failing_payload_only_reddens_the_reporter_that_owns_it(
+ tmp_path, label, reporter, expect_red
+):
+ """The launcher writes ONE verdict for a kernel that now holds two
+ unrelated experiments. A reporter reading it directly would announce a
+ failure it cannot describe, over a section listing none, and point at the
+ wrong half of a 13-minute kernel."""
+ evidence = tmp_path / "evidence"
+ evidence.mkdir()
+ reports = [
+ {"label": "control", "passed": label != "control", "steps": []},
+ {"label": "studio-gpu", "passed": label != "studio-gpu", "assertions": []},
+ ]
+ (evidence / "launch_result.json").write_text(
+ json.dumps(
+ {
+ "verdict": "fail",
+ "reason": "1 of 2 payload(s) failed their assertions",
+ "slug": "u/s",
+ "kernel_state": "COMPLETE",
+ "reports": reports,
+ }
+ )
+ )
+ proc = subprocess.run(
+ [
+ sys.executable,
+ str(REPO_ROOT / ".github" / "scripts" / reporter / "report.py"),
+ "--evidence",
+ str(evidence),
+ "--expect",
+ "1",
+ ],
+ capture_output = True,
+ text = True,
+ )
+ assert (proc.returncode == 1) is expect_red, proc.stdout
+
+
+def test_the_build_step_actually_packs_studio_in():
+ """The whole pipelining claim in this workflow's header rests on one flag.
+
+ Without it the kernel builds four legs, every reporter still renders, the
+ Studio section reads NOT RUN with a plausible-sounding reason, and the job
+ is green -- which is indistinguishable from a run whose sampling declined.
+ """
+ source = NOTEBOOK_WORKFLOW.read_text(encoding = "utf-8")
+ build = source.split("- name: Build the kernel notebooks")[1].split("- name:")[0]
+ assert "--with-studio" in build
+ assert "--studio-args" in build
+
+
+# ------------------------------------------------------------- the prefetch lane
+
+
+def test_the_prefetch_lane_never_takes_a_card(tmp_path):
+ """It is CPU and network work, and a card it held would be a card idle.
+
+ The whole saving is that downloading happens BESIDE training rather than
+ in front of it. A prefetch that consumed a GPU slot would move the wait
+ rather than remove it, and would also break the packing arithmetic that
+ assumes exactly two lanes compete for two cards.
+ """
+ hub = _HubStub()
+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2, prefetch_repos = ("a/big", "b/small"), hub = hub)
+ assert driven["stood_down"] is None
+ assert hub.calls == ["a/big", "b/small"], hub.calls
+ # Every papermill call is a LEG. The prefetch is not one of them, so it
+ # cannot have been handed CUDA_VISIBLE_DEVICES.
+ assert len(driven["stub"].papermill) == len(ALL_LEGS), driven["stub"].papermill
+ # Two legs on a card is legal now (see the VRAM budget); what must hold is
+ # that the summed appetite never exceeds what the card has.
+ for card, peak in driven["stub"].peak_card_gb.items():
+ assert peak <= 13.0, (card, peak, driven["stub"].same_card_overlaps)
+
+
+def test_the_leg_prefetch_does_not_redirect_hf_home(tmp_path):
+ """The legs read the Kaggle image's DEFAULT cache.
+
+ Pointing the lane at a private root is the silent failure this guards: it
+ downloads all 12 GB perfectly, into a directory no leg looks in, reports
+ success, and the run is green and no faster. Nothing at runtime would say
+ so, which is why it is asserted here.
+ """
+ hub = _HubStub()
+ before = os.environ.get("HF_HOME")
+ _drive_packed(tmp_path, ALL_LEGS, gpus = 2, prefetch_repos = ("a/big",), hub = hub)
+ assert hub.hf_home_at_call == [before], hub.hf_home_at_call
+ assert os.environ.get("HF_HOME") == before
+
+
+def test_a_failing_prefetch_does_not_fail_the_kernel(tmp_path):
+ """Graceful degradation is the entire safety argument for shipping this.
+
+ The leg that wants the model still downloads it itself, exactly as it did
+ before the lane existed, so a prefetch failure costs seconds. If it could
+ fail the kernel it would be a brand new way to go red for something that
+ is not under test -- on a payload that is not even the subject of the run.
+ """
+ hub = _HubStub(fail_for = ("a/big", "b/small"))
+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2, prefetch_repos = ("a/big", "b/small"), hub = hub)
+ assert driven["stood_down"] is None
+ assert len(driven["stub"].papermill) == len(ALL_LEGS), driven["stub"].papermill
+ assert all(r.get("returncode") == 0 for r in driven["results"].values()), driven["results"]
+
+
+def test_no_prefetch_repos_leaves_the_schedule_exactly_as_it_was(tmp_path):
+ """The lane is opt-in at the call site, and off means OFF: no thread, no
+ huggingface_hub import, no behaviour change for a kernel built without it."""
+ hub = _HubStub()
+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2, prefetch_repos = (), hub = hub)
+ assert hub.calls == [], hub.calls
+ assert driven["stood_down"] is None
+ assert len(driven["stub"].papermill) == len(ALL_LEGS)
+
+
+def test_the_prefetch_list_matches_the_models_the_legs_actually_load():
+ """A prefetch of the wrong repo downloads happily and warms nothing.
+
+ There is no runtime feedback for this: the lane reports success, the legs
+ download for themselves, and the only symptom is a saving that never
+ arrives. So the declared list is checked against the DEFAULT_MODEL the
+ payload scripts really carry, read out of their source.
+
+ The DEFAULT_MODEL is where an earlier version of this test stopped, and
+ stopping there is what let the bug through. What a leg ASKS FOR and what it
+ LOADS are different for gpt-oss: `unsloth/gpt-oss-20b` is MXFP4, sm_75
+ cannot read MXFP4, and unsloth redirects to `-unsloth-bnb-4bit` at load
+ time. The old assertion compared the prefetch list against the declared
+ name, so it agreed with a prefetch of 55.1 GB that no leg ever opened. It
+ now applies LOAD_REDIRECTS first, and separately pins that the redirect it
+ is applying is the one the payload actually documents.
+ """
+ from legs import LEGS, LOAD_REDIRECTS, PREFETCH_REPOS
+
+ # Reading DEFAULT_MODEL out of two named scripts is where this stopped, and
+ # it was too narrow twice over: it knew nothing of legs that pass --model on
+ # the command line, and nothing of payloads beyond the two it names. The set
+ # equality it then asserted was therefore satisfied by a prefetch list
+ # missing the leg that SETS the makespan. The walk lives in
+ # test_prefetch_covers_the_wired_legs.py, which owns the rule; this file
+ # keeps the redirect-provenance half below, which that one does not cover.
+ from test_prefetch_covers_the_wired_legs import models_for
+
+ loaded = set()
+ for leg in LEGS.values():
+ loaded |= models_for(leg)
+ stray = sorted(set(PREFETCH_REPOS) - loaded)
+ assert not stray, f"prefetching {stray}, which no leg loads even after LOAD_REDIRECTS"
+
+ # LOAD_REDIRECTS is only as good as its agreement with reality. If a
+ # redirect stops being real, this list must stop claiming it -- or the
+ # prefetch goes back to warming a cache nobody reads, in the other
+ # direction and just as invisibly.
+ #
+ # There are two kinds of evidence for a redirect and they need different
+ # checks. gpt-oss's target is written in the payload, so it can be read out
+ # of the source. The Qwen one cannot: unsloth derives it at load time
+ # through FLOAT_TO_INT_MAPPER and no file in this repo names it. Its only
+ # evidence is a resolved_checkpoint field in a kernel report. So a redirect
+ # no payload mentions must CITE the kernel that measured it, which is
+ # checkable, rather than being taken on trust.
+ sources = "".join(path.read_text(encoding = "utf-8") for path in sorted(SMOKE_DIR.glob("*.py")))
+ legs_src = (Path(build_kernel.__file__).parent / "legs.py").read_text(encoding = "utf-8")
+ for declared, actual in LOAD_REDIRECTS.items():
+ if actual in sources:
+ continue
+ assert re.search(rf"#[^\n]*{re.escape(actual)}", legs_src) or any(
+ actual in line and line.lstrip().startswith("#") for line in legs_src.splitlines()
+ ), (
+ f"LOAD_REDIRECTS says {declared} loads as {actual}, no payload "
+ f"mentions {actual}, and no comment in legs.py cites the run that "
+ f"measured it -- so the redirect is asserted and not observed"
+ )
+ # Qwen FIRST, and the reasoning inverted once the schedule was simulated
+ # end to end. gpt-oss is bigger, but it is wanted by ONE leg whose setup
+ # does not finish until ~160s anyway, whereas the small model gates THREE
+ # legs and costs ~20s. Fetching the big one first pushes the small one out
+ # past the moment the first leg is ready and delays three legs to give one
+ # a head start it did not need.
+ assert "Qwen" in PREFETCH_REPOS[0], PREFETCH_REPOS
+ assert "gpt-oss" in PREFETCH_REPOS[-1], PREFETCH_REPOS
+
+
+def test_the_generated_prefetch_cell_runs_not_merely_compiles():
+ """Compiling it is not enough, and that is not hypothetical here.
+
+ The first version interpolated `hf_home` with `json.dumps`, so `None`
+ became the JSON literal `null`. It compiled cleanly and died with a
+ NameError the first time it RAN -- which on the real thing means minutes
+ into a paid Kaggle session.
+ """
+ prefetch = build_kernel._prefetch_builder()
+ hub = _HubStub(hold = 0.0)
+ saved = sys.modules.get("huggingface_hub")
+ sys.modules["huggingface_hub"] = hub
+ try:
+ for hf_home in (None, "/tmp/somewhere"):
+ source = prefetch.prefetch_cell(
+ ["a/b"], hf_home = hf_home, attempt_timeout = 2, total_timeout = 5
+ )
+ exec(compile(source, "<prefetch>", "exec"), {"__name__": "prefetch"})
+ finally:
+ if saved is None:
+ sys.modules.pop("huggingface_hub", None)
+ else:
+ sys.modules["huggingface_hub"] = saved
+ assert hub.calls == ["a/b", "a/b"], hub.calls
+
+
+def test_a_repos_allow_patterns_reach_the_hub_and_a_bare_repo_stays_unfiltered():
+ """Computing the right glob and not passing it looks identical everywhere.
+
+ The patterns are worked out in the studio builder, carried through
+ `_normalise`, interpolated into generated source and echoed into the
+ summary. Every one of those steps can be right while the `snapshot_download`
+ call omits the keyword, and the only symptom is the 69.1 GB bill this was
+ written to stop -- the summary would still print the glob it meant to use.
+
+ The bare-string case is asserted alongside, because "filter everything"
+ breaks the opposite way: a small model whose every file is loaded must not
+ quietly acquire a filter and arrive incomplete.
+ """
+ prefetch = build_kernel._prefetch_builder()
+ hub = _HubStub(hold = 0.0)
+ saved = sys.modules.get("huggingface_hub")
+ sys.modules["huggingface_hub"] = hub
+ try:
+ source = prefetch.prefetch_cell(
+ [("big/gguf", ["*UD-Q4_K_XL*"]), "small/model"],
+ attempt_timeout = 2,
+ total_timeout = 5,
+ )
+ exec(compile(source, "<prefetch>", "exec"), {"__name__": "prefetch"})
+ finally:
+ if saved is None:
+ sys.modules.pop("huggingface_hub", None)
+ else:
+ sys.modules["huggingface_hub"] = saved
+
+ assert hub.calls == ["big/gguf", "small/model"], hub.calls
+ assert hub.patterns_at_call == [["*UD-Q4_K_XL*"], None], hub.patterns_at_call
+
+
+def test_the_last_prefetch_attempt_falls_back_to_classic_http():
+ """Retrying a STALLING transport is how a retry loop eats the session.
+
+ Xet retries 408/429/5xx itself with backoff (5 attempts, 3s base, a
+ six-minute cap per delay), so a throttled transfer can sit inside one call
+ for minutes without raising. Repeating the same transport inherits that.
+ The escalation to HF_HUB_DISABLE_XET is what makes the last attempt a
+ genuinely different thing to try.
+ """
+ prefetch = build_kernel._prefetch_builder()
+ seen: list = []
+
+ class _Recording(_HubStub):
+ def snapshot_download(
+ self,
+ repo_id = None,
+ **kw,
+ ):
+ seen.append(os.environ.get("HF_HUB_DISABLE_XET"))
+ raise RuntimeError("always")
+
+ saved = sys.modules.get("huggingface_hub")
+ before = os.environ.get("HF_HUB_DISABLE_XET")
+ sys.modules["huggingface_hub"] = _Recording(hold = 0.0)
+ try:
+ source = prefetch.prefetch_cell(["a/b"], attempt_timeout = 1, total_timeout = 30)
+ exec(compile(source, "<prefetch>", "exec"), {"__name__": "prefetch"})
+ finally:
+ if saved is None:
+ sys.modules.pop("huggingface_hub", None)
+ else:
+ sys.modules["huggingface_hub"] = saved
+ assert seen[-1] == "1", seen
+ assert seen[:-1] == [None] * (len(seen) - 1), seen
+ # ...and it is not left set for whatever runs next in this interpreter.
+ assert os.environ.get("HF_HUB_DISABLE_XET") == before
+
+
+def test_the_studio_prefetch_lands_in_studios_own_cache():
+ """Studio keeps a private HF_HOME, and the prefetch must follow it there.
+
+ The t4 lane deliberately does the opposite -- it leaves HF_HOME alone so
+ the training legs can read what it warms -- so the two are easy to conflate
+ and the failure is silent either way: bytes land somewhere real, the
+ download reports success, and the payload that wanted them downloads again.
+
+ The install cell passes hf_home=None to INHERIT, which is only correct
+ because the setup cell has already exported Studio's root. That ordering is
+ what is pinned here; a prefetch cell hoisted above setup would inherit the
+ image default and quietly stop helping.
+ """
+ studio = build_kernel._studio_builder()
+ notebook = studio.build_payload_notebook(
+ unsloth_ref = "x",
+ repo_url = "https://h/r",
+ payload_args = "--max-steps 8",
+ phase = "install",
+ )
+ sources = ["".join(cell["source"]) for cell in notebook["cells"]]
+ sets_home = [i for i, src in enumerate(sources) if 'os.environ["HF_HOME"]' in src]
+ prefetches = [i for i, src in enumerate(sources) if "KAGGLE_CI_PREFETCH" in src]
+ assert sets_home, "the install phase never exports Studio's HF_HOME"
+ assert prefetches, "the install phase carries no prefetch"
+ assert min(sets_home) < min(prefetches), (
+ f"HF_HOME is exported at cell {min(sets_home)} but the prefetch runs at "
+ f"{min(prefetches)}, so it would warm the image default instead"
+ )
+ # And it must not hardcode a root of its own alongside the inherited one.
+ assert "_HF_HOME = None" in sources[min(prefetches)], sources[min(prefetches)][:400]
+
+
+def test_the_studio_prefetch_follows_the_dispatched_models():
+ """--chat-model and --train-model are dispatch inputs.
+
+ A hardcoded pair here would prefetch the defaults while the payload loaded
+ something else -- which downloads happily, warms a cache nobody reads, and
+ reports success.
+
+ --chat-variant is read for the same reason and now matters as much. Studio
+ loads ONE quant from a GGUF repo that ships many, so an unfiltered snapshot
+ is not merely generous: run 32667451396 pulled 69.1 GB of Qwen3.5-2B-GGUF
+ to serve a single UD-Q4_K_XL file, and on a 4-core Kaggle box that CPU came
+ straight out of the payloads the prefetch exists to speed up.
+ """
+ studio = build_kernel._studio_builder()
+ chat, train = studio._models_from("--chat-model a/b --train-model c/d")
+ assert chat == ("a/b", ["*UD-Q4_K_XL*"]), chat
+ assert train == "c/d", train
+ assert studio._models_from("--chat-model=e/f")[0][0] == "e/f"
+
+ # The filter follows the dispatched variant rather than the default, or a
+ # run that overrode it would prefetch a quant it never loads.
+ picked, patterns = studio._models_from("--chat-variant Q8_0")[0]
+ assert patterns == ["*Q8_0*"], patterns
+
+ # Loose at BOTH ends on purpose. A split GGUF is named
+ # `...UD-Q4_K_XL-00001-of-00002.gguf`, so a suffix-anchored glob would
+ # match the single-file case and miss every shard of the split one --
+ # downloading nothing, reporting success, leaving Studio to fetch it.
+ assert patterns[0].startswith("*") and patterns[0].endswith("*"), patterns
+
+ defaults = studio._models_from("--max-steps 8")
+ flat = [entry[0] if isinstance(entry, tuple) else entry for entry in defaults]
+ payload = (SMOKE_DIR.parent / "studio_gpu" / "run_studio_gpu.py").read_text(encoding = "utf-8")
+ for flag in ("--chat-model", "--train-model", "--chat-variant"):
+ declared = re.search(rf'ap\.add_argument\("{flag}", default = "([^"]+)"\)', payload)
+ assert declared, f"{flag} default not found in run_studio_gpu.py"
+ if flag == "--chat-variant":
+ assert defaults[0][1] == [f"*{declared.group(1)}*"], defaults[0]
+ else:
+ assert declared.group(1) in flat, (declared.group(1), flat)
+
+
+def test_the_report_shows_what_the_prefetch_achieved(tmp_path):
+ """The number the leg order is arranged around has to be readable without
+ downloading an artifact -- including when it says the lane did not help."""
+ import report as t4_report
+
+ evidence = tmp_path / "evidence"
+ evidence.mkdir()
+ (evidence / "kernel.log").write_text(
+ 'KAGGLE_CI_PREFETCH {"repo": "unsloth/gpt-oss-20b", "ok": true, "seconds": 141.0, '
+ '"download_seconds": 141.0, "bytes": 12000000000, "mb_per_s": 85.1, '
+ '"transport": "auto", "attempts": 1}\n'
+ 'KAGGLE_CI_PREFETCH {"repo": "unsloth/Qwen2.5-0.5B-Instruct", "ok": false, '
+ '"seconds": 9.0, "download_seconds": null, "bytes": 0, "mb_per_s": null, '
+ '"transport": "http", "attempts": 3, "error": "nope"}\n',
+ encoding = "utf-8",
+ )
+ lines = "\n".join(t4_report.prefetch_table(evidence))
+ assert "unsloth/gpt-oss-20b" in lines
+ assert "141.0" in lines and "85.1" in lines and "12.0" in lines
+ assert "**NO**" in lines, "a failed prefetch must be visible, not rounded away"
+ assert "fallback" in lines, "a failed prefetch must say what it costs the schedule"
+ # A kernel built without the lane gets no section at all, rather than a
+ # table of zeroes that reads like a lane that ran and achieved nothing.
+ bare = tmp_path / "bare"
+ bare.mkdir()
+ (bare / "kernel.log").write_text("nothing to see", encoding = "utf-8")
+ assert t4_report.prefetch_table(bare) == []
+
+ # ...and it is WIRED IN. Calling the renderer directly proves it renders,
+ # which is not the same claim: deleting the one line that appends it to the
+ # summary left this test green, because a table nobody calls still formats
+ # perfectly. So drive main() and read what a human would actually see.
+ (evidence / "launch_result.json").write_text(
+ json.dumps(
+ {
+ "verdict": "pass",
+ "reason": "all 1 payload(s) passed",
+ "slug": "u/s",
+ "kernel_state": "COMPLETE",
+ "reports": [{"label": "control", "passed": True, "steps": []}],
+ }
+ ),
+ encoding = "utf-8",
+ )
+ summary = tmp_path / "summary.md"
+ proc = subprocess.run(
+ [sys.executable, str(CI_DIR / "report.py"), "--evidence", str(evidence), "--expect", "1"],
+ capture_output = True,
+ text = True,
+ env = {**os.environ, "GITHUB_STEP_SUMMARY": str(summary)},
+ )
+ assert proc.returncode == 0, proc.stdout
+ rendered = summary.read_text(encoding = "utf-8")
+ assert "model prefetch" in rendered, rendered
+ assert "unsloth/gpt-oss-20b" in rendered, rendered
+
+
+def test_gptoss_never_shares_a_card(tmp_path):
+ """12.78 GB of a 14.56 GB card, so it is alone by arithmetic.
+
+ Not by a special case -- there is no `if name == "gptoss"` anywhere. If a
+ leg's appetite ever grows past the budget it stops sharing on its own, and
+ if gptoss ever shrinks it starts sharing on its own. What must never happen
+ is the pairing that put 13.48 GB on a card and came back as an OOM reading
+ like a code failure.
+ """
+ durations = {f"t4_{n}.ipynb": 0.4 for n in ALL_LEGS}
+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2, durations = durations)
+ for card, together in driven["stub"].same_card_overlaps:
+ assert "t4_gptoss.ipynb" not in together, (card, together)
+
+
+def test_two_small_legs_do_share_a_card(tmp_path):
+ """The feature, asserted positively.
+
+ Every other guard here is a bound -- never over budget, never more than two
+ -- and every one of them is satisfied by a scheduler that co-schedules
+ NOTHING. Without this the whole change could silently do no work at all and
+ the suite would stay green.
+ """
+ durations = {f"t4_{n}.ipynb": 0.4 for n in ALL_LEGS}
+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2, durations = durations)
+ assert driven[
+ "stub"
+ ].same_card_overlaps, "no card ever held two legs at once, so the VRAM budget bought nothing"
+ assert max(driven["stub"].peak_card_legs.values()) == 2
+ # The admission ledger has to balance. A leg that reserves and never
+ # releases leaks capacity, and with four legs and four worker slots nothing
+ # ever waits, so the leak is invisible until the day a fifth leg is wired
+ # and one card silently stops taking work.
+ assert driven["card_load"] and all(abs(v) < 1e-9 for v in driven["card_load"].values()), driven[
+ "card_load"
+ ]
+ assert all(v == 0 for v in driven["card_count"].values()), driven["card_count"]
+
+
+def test_the_declared_vram_matches_what_the_legs_reported():
+ """`Leg.vram_gb` decides who may share a card, and nothing checks it at
+ runtime: a leg that under-declares gets admitted beside another and the
+ contention comes back as an OOM attributed to whichever leg happened to
+ allocate last.
+
+ So the declared figures are checked against the peaks the payloads really
+ reported, captured in the evidence of run 32611343797 and committed beside
+ this test.
+ """
+ measured = json.loads(
+ (Path(__file__).parent / "t4_smoke" / "measured_vram.json").read_text(encoding = "utf-8")
+ )
+ for name, peak in measured["peak_reserved_gb"].items():
+ declared = LEGS[name].vram_gb
+ assert declared >= peak, (
+ f"{name} declares {declared} GB but peaked at {peak} GB, so the "
+ "admission check would let something share a card with it that "
+ "does not fit"
+ )
+ # ...and not so far above it that the budget stops admitting anything.
+ assert declared <= peak + 1.5, (name, declared, peak)
+ assert measured["card_total_gb"] > 13.0, measured
+
+
+# ------------------------------------------------- Studio sharing a card
+
+
+def test_studio_waits_for_the_queue_by_default(tmp_path, monkeypatch):
+ """The default keeps both T4s visible to Studio.
+
+ Sharing is faster and narrower, so it must be something someone turned on,
+ not something that arrived with an unrelated change.
+ """
+ monkeypatch.setenv("CUDA_VISIBLE_DEVICES", AMBIENT_CUDA)
+ durations = {f"t4_{n}.ipynb": 0.4 for n in ALL_LEGS}
+ durations[STUDIO_INSTALL] = 0.1
+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2, studio = STUDIO, durations = durations)
+ calls = {c["notebook"]: c for c in driven["stub"].papermill}
+ assert calls[STUDIO_TEST]["cuda"] == AMBIENT_CUDA, calls[STUDIO_TEST]
+ legs = [
+ (c["notebook"], c["cuda"])
+ for c in driven["stub"].papermill
+ if c["notebook"].startswith("t4_")
+ ]
+ assert len(legs) == len(ALL_LEGS)
+
+
+def test_studio_concurrent_takes_a_card_gptoss_is_not_on(tmp_path):
+ """--studio-concurrent trades coverage for time and must pay honestly.
+
+ Two things have to hold. Studio is PINNED to a card, because sharing means
+ it is no longer choosing between two. And it is admitted by the same VRAM
+ check the legs use, so it can never land beside gptoss: 12.78 + 2.2 is
+ 14.98 on a card budgeted to 13.0, which is the pairing that came back as an
+ OOM reading like a code failure.
+
+ Driven with gptoss as the ONLY leg, so the placement is deterministic
+ rather than a race between the stub's durations. An earlier version used
+ all four legs and asserted on recorded overlaps; the small legs finished
+ while Studio was still building its venv, so no overlap was ever recorded
+ and deleting the VRAM check left the test green.
+ """
+ driven = _drive_packed(
+ tmp_path,
+ ["gptoss"],
+ gpus = 2,
+ studio = STUDIO,
+ durations = {"t4_gptoss.ipynb": 4.0, STUDIO_INSTALL: 0.05},
+ after_gpu_concurrent = True,
+ )
+ calls = {c["notebook"]: c for c in driven["stub"].papermill}
+ assert STUDIO_TEST in calls, sorted(calls)
+ assert calls[STUDIO_TEST]["cuda"] in ("0", "1"), calls[STUDIO_TEST]
+ assert calls["t4_gptoss.ipynb"]["cuda"] in ("0", "1"), calls["t4_gptoss.ipynb"]
+ assert calls[STUDIO_TEST]["cuda"] != calls["t4_gptoss.ipynb"]["cuda"], (
+ f"Studio was put on the same card as gptoss "
+ f"({calls[STUDIO_TEST]['cuda']}): 12.78 + 2.2 GB on a 13.0 GB budget"
+ )
+ for card, peak in driven["stub"].peak_card_gb.items():
+ assert peak <= 13.0, (card, peak, driven["stub"].same_card_overlaps)
+
+
+def test_studio_concurrent_still_skips_when_its_install_failed(tmp_path):
+ """The dependency survives the faster path.
+
+ Running the assertions against a half-built tree fails on a missing venv,
+ which reads like the code under test broke rather than like the install
+ did -- and on this path the test half is started from its own thread, so
+ the gate had to be re-implemented rather than inherited.
+ """
+
+ class _InstallFails(_PackedStub):
+ def run(self, cmd, **kw):
+ cmd = [str(c) for c in cmd]
+ if "papermill" in cmd and STUDIO_INSTALL in " ".join(cmd):
+ self.papermill.append(
+ {
+ "notebook": STUDIO_INSTALL,
+ "cuda": None,
+ "kernel": None,
+ "compile_location": None,
+ }
+ )
+ Path(cmd[cmd.index("papermill") + 2]).write_text("{}", encoding = "utf-8")
+ return types.SimpleNamespace(returncode = 1, stdout = "", stderr = "")
+ return super().run(cmd, **kw)
+
+ driver = build_kernel.build_kernel(
+ SMOKE_DIR,
+ ALL_LEGS,
+ unsloth_ref = "main",
+ zoo_ref = "main",
+ extra_args = (),
+ per_run_timeout = 60,
+ skip_reference = True,
+ studio = STUDIO,
+ after_gpu_concurrent = True,
+ )
+ stub = _InstallFails(gpus = 2, durations = {f"t4_{n}.ipynb": 0.3 for n in ALL_LEGS})
+ stub.root = tmp_path
+ stub.venv_root = tmp_path / "venvs"
+ saved = sys.modules["subprocess"]
+ sys.modules["subprocess"] = stub
+ namespace: dict = {}
+ try:
+ for cell in driver["cells"][:2]:
+ source = (
+ "".join(cell["source"])
+ .replace("/tmp/t4ci_venvs", str(tmp_path / "venvs"))
+ .replace("/kaggle/working", str(tmp_path))
+ )
+ exec(compile(source, "<driver-cell>", "exec"), namespace)
+ finally:
+ sys.modules["subprocess"] = saved
+ results = namespace.get("results") or {}
+ assert STUDIO_TEST in results, sorted(results)
+ assert results[STUDIO_TEST]["returncode"] is None, results[STUDIO_TEST]
+ assert "install lane did not succeed" in results[STUDIO_TEST]["error"]
+ assert STUDIO_TEST not in [c["notebook"] for c in stub.papermill]
+
+
+def test_a_legs_overlay_reaches_its_payload_and_never_carries_torch(tmp_path):
+ """The overlay must WIN over the venv, and must not bring native packages.
+
+ Two failures are being guarded, and they look identical from outside:
+
+ * An overlay built but never put on ``PYTHONPATH``. The leg runs on the base
+ versions, trains, passes, and reports a version table nobody reads. This
+ is the reason the payload's env is inspected rather than the fact that a
+ ``pip install --target`` happened.
+ * An overlay that shadows torch. ``pip install --dry-run --report`` resolves
+ the FULL closure, and a closure containing transformers frequently
+ contains torch too; installing that into the overlay puts a second torch
+ ahead of the one already loaded against this box's CUDA runtime. The stub
+ resolver therefore returns torch on purpose, so a driver that forgot to
+ filter fails here instead of on a Kaggle session.
+
+ Measured basis for the mechanism: kernel unsloth-probe-overlay-t4-r2-38ac4d
+ on a real T4 resolved transformers==4.57.6 + trl~=0.22.0 to three packages,
+ 115.9 MB, in 10.0s, with transformers and trl imported from the overlay and
+ torch still from the base.
+ """
+ leg = "canary"
+ overlay = ("transformers==4.57.6", "trl~=0.22.0")
+ original = LEGS[leg].overlay
+ object.__setattr__(LEGS[leg], "overlay", overlay)
+ try:
+ stub = _drive_packed(tmp_path, [leg], gpus = 2)["stub"]
+ finally:
+ object.__setattr__(LEGS[leg], "overlay", original)
+
+ installs = [c for c in stub.overlay_installs if "--target" in c]
+ assert installs, "the leg declared an overlay and nothing was installed into one"
+ target = installs[0][installs[0].index("--target") + 1]
+ assert f"overlay_t4_{leg}" in target, target
+
+ installed = " ".join(installs[0]).lower()
+ assert "transformers==4.57.6" in installed, installed
+ assert (
+ "torch==" not in installed
+ ), f"the overlay installed torch, which shadows the base one: {installed}"
+
+ record = [p for p in stub.papermill if p["notebook"] == f"t4_{leg}.ipynb"]
+ assert record, [p["notebook"] for p in stub.papermill]
+ pythonpath = record[0]["env"].get("PYTHONPATH", "")
+ assert target in pythonpath.split(os.pathsep), (
+ f"the overlay was built at {target} but the payload's PYTHONPATH is "
+ f"{pythonpath!r}, so the child would import the base versions"
+ )
+
+
+def test_a_leg_with_no_overlay_gets_no_pythonpath(tmp_path):
+ """The control case, without which the test above proves only that a
+ variable exists somewhere.
+
+ A driver that unconditionally set PYTHONPATH -- to the overlay root, to an
+ empty directory, to anything -- would satisfy the first guard while giving
+ every leg the same environment. The legs' whole purpose is that they differ.
+ """
+ stub = _drive_packed(tmp_path, ["control"], gpus = 2)["stub"]
+ assert not [
+ c for c in stub.overlay_installs if "--target" in c
+ ], "a leg declaring no overlay had one built for it"
+ record = [p for p in stub.papermill if p["notebook"] == "t4_control.ipynb"]
+ assert record
+ assert "overlay_" not in record[0]["env"].get("PYTHONPATH", "")
+
+
+def test_every_leg_installs_bitsandbytes_and_probes_that_it_imports():
+ """bitsandbytes has to be asked for, and asked for EARLY.
+
+ It is absent from every dependency set the CI resolves: `unsloth_zoo`
+ declares 57 requirements and bitsandbytes is not among them, and git-main
+ `unsloth` declares only seven unconditional dependencies (typer, rich,
+ pydantic, pyyaml, nest-asyncio, structlog, click) with bitsandbytes reachable
+ only through its CUDA extras. The released PyPI package DOES carry it
+ unconditionally, which is why notebooks installing from PyPI never notice --
+ and why this CI, which installs from git SHAs, must ask.
+
+ Without it the run gets a long way before failing: the install succeeds, the
+ model downloads, and it dies inside `from_pretrained` at
+ unsloth_zoo/patching_utils.py:386. Probing it in the import cell turns that
+ into a failure before the session is spent, which is the whole point of the
+ probe list.
+ """
+ for name, leg in LEGS.items():
+ flat = [spec for group in leg.install for spec in group]
+ assert any("bitsandbytes" in spec for spec in flat), (
+ f"leg {name!r} never installs bitsandbytes; on the Kaggle image it "
+ f"would fail inside from_pretrained after the model download"
+ )
+ assert "bitsandbytes" in leg.imports, (
+ f"leg {name!r} does not probe bitsandbytes, so a broken or missing "
+ f"copy surfaces minutes later as a model-loading error"
+ )
diff --git a/tests/kaggle/test_t4_payload_assertions.py b/tests/kaggle/test_t4_payload_assertions.py
index 2532b9fb086..4c3629e45cc 100644
--- a/tests/kaggle/test_t4_payload_assertions.py
+++ b/tests/kaggle/test_t4_payload_assertions.py
@@ -15,6 +15,7 @@ costs a Kaggle session has to be checkable without one.
from __future__ import annotations
import json
+import re
import sys
import types
from pathlib import Path
@@ -908,6 +909,26 @@ def test_the_committed_reference_pins_the_model_it_was_captured_on():
# -------------------------------------------- run_t4_smoke.py: main() paths
+def _batch_record(**over):
+ """A healthy batched-generation record, with fields overridable per test."""
+ base = {
+ "prompt_token_lengths": [11, 14, 9, 17, 12, 20, 8, 15],
+ "distinct_lengths": 8,
+ "padding_side_observed": "left",
+ "padding_side_after": "left",
+ "singles": [f"out{i}" for i in range(8)],
+ "batched": {
+ "2": [f"out{i}" for i in range(8)],
+ "4": [f"out{i}" for i in range(8)],
+ "8": [f"out{i}" for i in range(8)],
+ },
+ "agrees": {"2": True, "4": True, "8": True},
+ "empty_outputs": [],
+ }
+ base.update(over)
+ return base
+
+
def _cycle(index: int, losses: list[float]) -> dict:
return {
"run_index": index,
@@ -916,6 +937,13 @@ def _cycle(index: int, losses: list[float]) -> dict:
],
"generated": "__UNSLOTH__!!!",
"canary_found": True,
+ # A simulated HEALTHY cycle has to look healthy in every respect a real
+ # one is judged on, this record included. Leaving it out made
+ # `--check-batched-generation` (on by default, because a leg that
+ # quietly skips it stops covering #3699/#1066/#1456/#2138) fail every
+ # simulated run with "batched generation was never run" -- which is the
+ # rule working, on a fixture that had not kept up.
+ "batched_generation": _batch_record(),
"adapter_files": ["adapter_config.json", "adapter_model.safetensors"],
"saved_adapter": _adapter_state(),
"determinism": {},
@@ -1773,3 +1801,569 @@ def test_gptoss_still_reads_a_bf16_card_and_a_t4_the_way_it_did():
)
== []
)
+
+
+def test_batched_generation_runs_end_to_end_against_a_stub_model():
+ """Every other test in this section feeds `batched_generation_failures` a
+ dict someone typed. That checks the RULE and never once executes the code
+ that produces the dict, which is how kernel unsloth-probe-defaultleg-723c28
+ trained all ten steps on a real T4 and then died on
+
+ NameError: name 'torch' is not defined
+
+ inside `batched_generation` itself. Every torch user in run_t4_smoke.py
+ imports it inside the function; that one did not, and no CPU test noticed
+ because none of them ever called it.
+
+ So drive the real function with a stub tokenizer and model. The stub echoes
+ a deterministic continuation per row, so agreement across batch sizes is
+ guaranteed and this asserts the plumbing rather than the model: shapes,
+ the padded-width slice, and that the function runs at all.
+ """
+ import torch
+
+ from run_t4_smoke import batched_generation, batched_generation_failures
+
+ class _Enc(dict):
+ def to(self, _device):
+ return self
+
+ class _Tok:
+ padding_side = "right"
+ pad_token = "<pad>"
+ eos_token = "<eos>"
+ pad_token_id = 0
+
+ def __call__(
+ self,
+ text,
+ return_tensors = None,
+ padding = False,
+ ):
+ texts = [text] if isinstance(text, str) else list(text)
+ # One token per character, so a length spread in the prompts is a
+ # length spread in the ids and the padding is real.
+ ids = [[ord(c) % 100 + 1 for c in t] for t in texts]
+ if return_tensors is None:
+ return {"input_ids": ids[0] if isinstance(text, str) else ids}
+ width = max(len(i) for i in ids)
+ # LEFT padding, which is what the function asks for and what the
+ # padded-width slice below depends on.
+ padded = [[0] * (width - len(i)) + i for i in ids]
+ return _Enc(
+ input_ids = torch.tensor(padded),
+ attention_mask = torch.tensor([[0] * (width - len(i)) + [1] * len(i) for i in ids]),
+ )
+
+ def decode(
+ self,
+ row,
+ skip_special_tokens = False,
+ ):
+ return "".join(chr(int(v)) for v in row if int(v) != 0)
+
+ class _Model:
+ device = "cpu"
+
+ def generate(
+ self,
+ input_ids = None,
+ attention_mask = None,
+ max_new_tokens = 8,
+ **_kw,
+ ):
+ # Append the same continuation to every row, derived from that
+ # row's own unpadded content, so batching cannot change it.
+ outs = []
+ for row in input_ids:
+ real = [int(v) for v in row if int(v) != 0]
+ tail = [(sum(real) % 26) + 65] * max_new_tokens
+ outs.append([int(v) for v in row] + tail)
+ return torch.tensor(outs)
+
+ # Eight, because BATCH_SIZES tops out at 8 and the rule rejects a record
+ # whose largest batch could never have been formed. Lengths 1..8 so every
+ # batch pads and the padded-width slice is exercised rather than skipped.
+ prompts = ["abcdefgh"[:n] * 1 for n in range(1, 9)]
+ record = batched_generation(_Model(), _Tok(), prompts, max_new_tokens = 4)
+
+ assert record["distinct_lengths"] == 8, "the stub prompts must actually pad"
+ assert record["padding_side_observed"] == "left"
+ assert len(record["singles"]) == len(prompts)
+ assert all(s for s in record["singles"]), "the padded-width slice dropped everything"
+ assert batched_generation_failures(record) == []
+
+
+def test_a_healthy_batched_generation_record_reports_no_failures():
+ from run_t4_smoke import batched_generation_failures
+ assert batched_generation_failures(_batch_record()) == []
+
+
+def test_a_batch_whose_prompts_are_all_one_length_is_reported_as_proving_nothing():
+ from run_t4_smoke import batched_generation_failures
+
+ """The vacuity case, and the reason this check exists in this shape.
+
+ Left padding only happens when the prompts in a batch differ in length. Feed
+ eight identical-length prompts and every batch agrees with every single --
+ perfectly, every time, having never padded once. That is a green
+ left-padding assertion covering nothing, which is the failure mode this
+ whole file keeps being caught by.
+ """
+ failures = batched_generation_failures(
+ _batch_record(prompt_token_lengths = [12] * 8, distinct_lengths = 1)
+ )
+ assert any("nothing was ever padded" in f for f in failures), failures
+
+
+def test_a_padding_side_silently_flipped_to_right_is_a_failure():
+ from run_t4_smoke import batched_generation_failures
+
+ """#2138: a release forced the tokenizer padding side to right in inference.
+
+ Checked twice, before and after generating, because the override that
+ caused it happened INSIDE the inference path -- so the value this payload
+ set is not evidence of the value that was used.
+ """
+ assert any(
+ "padding_side_after" in f
+ for f in batched_generation_failures(_batch_record(padding_side_after = "right"))
+ )
+ assert any(
+ "padding_side_observed" in f
+ for f in batched_generation_failures(_batch_record(padding_side_observed = "right"))
+ )
+
+
+def test_a_batch_that_disagrees_with_one_at_a_time_is_a_failure():
+ from run_t4_smoke import batched_generation_failures
+
+ """#3699 / #1456: batched output diverging from sequential greedy output."""
+ broken = _batch_record()
+ broken["batched"]["4"] = ["different"] * 8
+ broken["agrees"]["4"] = False
+ failures = batched_generation_failures(broken)
+ assert any("batch size 4 did not reproduce" in f for f in failures), failures
+
+
+def test_empty_generations_are_a_failure_even_when_every_batch_agrees():
+ from run_t4_smoke import batched_generation_failures
+
+ """#1066: gibberish, of which "nothing at all" is the degenerate case.
+
+ Agreement alone is not health: a model that emits an empty string for every
+ prompt agrees with itself at every batch size.
+ """
+ failures = batched_generation_failures(
+ _batch_record(
+ singles = [""] * 8,
+ empty_outputs = [0, 1, 2, 3, 4, 5, 6, 7],
+ batched = {"2": [""] * 8, "4": [""] * 8, "8": [""] * 8},
+ )
+ )
+ assert any("generated nothing at all" in f for f in failures), failures
+
+
+def test_too_few_prompts_to_fill_the_largest_batch_is_reported():
+ from run_t4_smoke import batched_generation_failures
+
+ """A batch size larger than the prompt list silently becomes one small batch."""
+ failures = batched_generation_failures(
+ _batch_record(singles = ["a", "b"], prompt_token_lengths = [5, 9], distinct_lengths = 2)
+ )
+ assert any("never actually formed" in f for f in failures), failures
+
+
+def test_a_missing_batched_record_is_a_failure_not_a_pass():
+ from run_t4_smoke import batched_generation_failures
+ """A leg where the check never ran must not look like a leg where it passed."""
+ assert batched_generation_failures(None) == ["batched generation was never run"]
+
+
+# ------------------------------------------------- run_grpo_t4.py: the reward
+
+
+def _completions(*lengths):
+ """GRPO hands reward functions a list of message lists."""
+ return [[{"role": "assistant", "content": "x" * n}] for n in lengths]
+
+
+def test_the_length_reward_still_discriminates_at_the_lengths_the_model_emits():
+ """The leg's only instrument, and it was broken in the least visible way.
+
+ `reward_length` was `min(len(t), 200) / 200.0` while its docstring claimed
+ to be "SENSITIVE to a group's diversity". Kernels
+ unsloth-probe-grpo-rep2-b03be8 and -rep3-bc3828 recorded completions of
+ 2534 to 3396 characters, so every completion scored exactly 1.0, every
+ group tied, and the leg failed with `reward_std was zero on every step`.
+ Two runs in three died on it.
+
+ A reward that saturates below the range the model actually occupies reads
+ as a broken generation path in the report. So assert discrimination at the
+ OBSERVED lengths, not at convenient small ones: the old function passes any
+ test written with 10- and 20-character completions.
+ """
+ sys.path.insert(0, str(SMOKE_DIR))
+ from run_grpo_t4 import reward_length
+
+ for a, b in ((2534, 3396), (3013, 3328), (2738, 2633)):
+ scores = reward_length(_completions(a, b))
+ assert scores[0] != scores[1], (
+ f"lengths {a} and {b} scored identically ({scores}); the reward has "
+ f"saturated in the range the model actually produces"
+ )
+
+
+def test_the_length_reward_is_strictly_increasing_and_never_saturates():
+ """The property that makes the test above hold for lengths nobody has seen
+ yet. A cap is a length at which this stops being true, which is exactly how
+ the previous version passed review."""
+ from run_grpo_t4 import reward_length
+
+ lengths = [0, 1, 50, 200, 1000, 3000, 10000, 100000]
+ scores = reward_length(_completions(*lengths))
+ assert all(x < y for x, y in zip(scores, scores[1:])), scores
+ assert all(0.0 <= s < 1.0 for s in scores), scores
+
+
+def test_two_identical_completions_still_tie_because_that_is_real():
+ """Guard against overcorrecting. A group whose completions are genuinely
+ identical SHOULD tie -- that is degenerate generation and the leg is right
+ to fail on it. The bug was ties between DIFFERENT completions."""
+ from run_grpo_t4 import reward_length
+
+ scores = reward_length(_completions(2534, 2534))
+ assert scores[0] == scores[1]
+
+
+# --------------------------------------------------- gguf_export.py
+
+
+def _gguf_record(**over):
+ """A healthy export, shaped like the one measured on
+ unsloth-probe-gguf-q8-peft-920e3e: the GGUF in the SIBLING directory."""
+ record = {
+ "save_dir": "/tmp/q8p",
+ "requested_quantization": "q8_0",
+ "ok": True,
+ "seconds": 40.6,
+ "ggufs": [
+ {
+ "path": "/tmp/q8p_gguf/qwen3-0.6b.Q8_0.gguf",
+ "mb": 609.8,
+ "found_in": "/tmp/q8p_gguf",
+ "suffix": "_gguf",
+ }
+ ],
+ }
+ record.update(over)
+ return record
+
+
+def test_a_healthy_gguf_export_reports_no_failures():
+ from gguf_export import export_failures
+ assert export_failures(_gguf_record(), accept_quantizations = ("q8_0",)) == []
+
+
+def test_an_export_that_reported_ok_but_wrote_no_gguf_is_a_failure():
+ """The trap this module exists for. save_pretrained_gguf writes the merged
+ safetensors into the directory it was given and the GGUF into a SIBLING, so
+ code that globs the directory it passed finds nothing, raises nothing, and
+ calls it a successful export."""
+ from gguf_export import export_failures
+
+ failures = export_failures(_gguf_record(ggufs = []), accept_quantizations = ("q8_0",))
+ assert failures and "no .gguf" in failures[0]
+ assert "_gguf sibling" in failures[0]
+
+
+def test_a_gguf_search_finds_the_file_in_the_sibling_directory(tmp_path):
+ """Executed against a real directory pair rather than a hand-written dict,
+ because the whole point is WHERE the file is."""
+ from gguf_export import find_ggufs
+
+ save = tmp_path / "out"
+ save.mkdir()
+ (save / "model.safetensors").write_bytes(b"0" * 2048)
+ sibling = tmp_path / "out_gguf"
+ sibling.mkdir()
+ (sibling / "qwen3-0.6b.Q8_0.gguf").write_bytes(b"0" * 4096)
+
+ found = find_ggufs(str(save))
+ assert len(found) == 1, found
+ assert found[0]["suffix"] == "_gguf"
+ assert found[0]["path"].endswith("qwen3-0.6b.Q8_0.gguf")
+
+
+def test_a_gguf_that_is_only_a_header_is_not_an_export():
+ from gguf_export import export_failures
+ failures = export_failures(
+ _gguf_record(
+ ggufs = [
+ {
+ "path": "/tmp/q8p_gguf/x.Q8_0.gguf",
+ "mb": 0.1,
+ "found_in": "/tmp/q8p_gguf",
+ "suffix": "_gguf",
+ }
+ ]
+ ),
+ accept_quantizations = ("q8_0",),
+ )
+ assert failures and "header and no weights" in failures[0]
+
+
+def test_a_model_allowed_to_override_the_quantization_still_passes():
+ """gpt-oss answers q8_0 with "GPT-OSS does not support GGUF quantization
+ (requested: q8_0). Overriding to MXFP4 format." That is documented
+ behaviour, so a leg that accepts MXFP4 must not fail on it."""
+ from gguf_export import export_failures
+
+ record = _gguf_record(
+ ggufs = [
+ {
+ "path": "/tmp/g_gguf/gpt-oss-20b.MXFP4.gguf",
+ "mb": 11800.0,
+ "found_in": "/tmp/g_gguf",
+ "suffix": "_gguf",
+ }
+ ]
+ )
+ assert export_failures(record, accept_quantizations = ("mxfp4",)) == []
+ # ... and a leg that does NOT accept it still says so.
+ failures = export_failures(record, accept_quantizations = ("q8_0",))
+ assert failures and "accepted quantization" in failures[0]
+
+
+def test_a_gguf_that_no_runner_could_execute_is_a_failure():
+ from gguf_export import run_failures
+ failures = run_failures(
+ {
+ "gguf": "/tmp/q8p_gguf/x.gguf",
+ "bench": {"seconds": 240.0, "error": "TimeoutExpired: ..."},
+ "completion": {"seconds": 240.0, "returncode": 1, "stderr": "bad magic"},
+ }
+ )
+ assert failures and "produced no output from any runner" in failures[0]
+
+
+def test_one_successful_runner_is_enough():
+ from gguf_export import run_failures
+ assert (
+ run_failures(
+ {
+ "gguf": "/tmp/q8p_gguf/x.gguf",
+ "bench": {"seconds": 12.0, "returncode": 0, "stdout": "tg128 ... 41.2"},
+ }
+ )
+ == []
+ )
+
+
+def test_a_bundle_with_no_runners_at_all_is_reported_as_that():
+ """Distinct from "the file does not run": a missing binary is a bundle
+ problem and blaming the model file for it would send the reader to the
+ wrong place."""
+ from gguf_export import run_failures
+
+ failures = run_failures(
+ {
+ "gguf": "/tmp/x.gguf",
+ "bench": {"skipped": "no llama-bench in the bundle"},
+ "completion": {"skipped": "no llama-completion in the bundle"},
+ }
+ )
+ assert failures and "missing from the llama.cpp bundle" in failures[0]
+
+
+def test_the_llama_cpp_facts_read_a_tuple_not_a_directory(tmp_path):
+ """install_llama_cpp returns (llama-quantize, convert_hf_to_gguf.py). An
+ earlier probe treated the return value as a bin directory and reported
+ "0 binaries", which was the probe being wrong, not the bundle being
+ empty."""
+ from gguf_export import llama_cpp_facts
+
+ quant = tmp_path / "llama-quantize"
+ conv = tmp_path / "convert_hf_to_gguf.py"
+ quant.write_text("")
+ conv.write_text("")
+
+ facts = llama_cpp_facts(
+ "Unsloth: Installing prebuilt llama.cpp b10472-mix-4b653db "
+ "(app-b10472-mix-4b653db-linux-x64-cpu.tar.gz) - skipping compilation.",
+ (str(quant), str(conv)),
+ )
+ assert facts["all_exist"] is True
+ assert facts["dir"] == str(tmp_path)
+ assert facts["prebuilt"] is True
+ assert facts["source_build_markers"] == []
+
+
+def test_a_source_build_is_visible_even_though_it_succeeds():
+ """The failure to catch is not "kernel missing" but "kernel built from
+ source": silent, correct, and many minutes on 4 vCPUs."""
+ from gguf_export import llama_cpp_facts
+
+ facts = llama_cpp_facts("-- Configuring done\ncmake --build . -j 4\n", ())
+ assert facts["prebuilt"] is False
+ assert "cmake" in facts["source_build_markers"]
+
+
+# ------------------------------- run_t4_smoke.py: parent -> child argv
+
+
+# Options the PARENT alone acts on, so their absence from the child command is
+# correct rather than a leak. Each is here for a stated reason; an entry added
+# without one is how this guard stops working.
+PARENT_ONLY_DESTS = {
+ "outdir", # the parent gives each cycle its own subdirectory
+ "cycle", # set by the parent per child, never forwarded verbatim
+ "repeat", # how many children to launch
+ "reference", # band check runs in the parent, over collected cycles
+ "rel_tol", # ... and its tolerances
+ "abs_floor",
+ "require_canary", # evaluated by the parent's failure collector
+ "check_batched_generation",
+ "export_gguf", # forwarded as a bare flag, asserted separately below
+ # The pin check reads the report the cycles produced, in the parent
+ # (run_t4_smoke.py:1741), so the children have nothing to do with it.
+ "pins",
+ # The plain-TRL control arm is spawned BY the parent, after the cycles, and
+ # ruled on there. A cycle child neither runs it nor judges it.
+ "compare_naive_trl",
+ "control_oom_is_ok",
+ # Kernel provenance IS collected in the child (it needs the loaded model),
+ # but the flag reaches it through the bare-flag block rather than the
+ # name/value pairs this check walks; asserted separately in
+ # test_kernel_provenance.py.
+ "kernel_provenance",
+ # The vision run is spawned BY the parent, after the cycles and in a
+ # process of its own: it loads a second model, and two 4bit models resident
+ # at once on a 14.56GB T4 is how a leg becomes an OOM blamed on the thing
+ # it was testing.
+ "vision_run",
+}
+
+
+def _smoke_source() -> str:
+ return (SMOKE_DIR / "run_t4_smoke.py").read_text(encoding = "utf-8")
+
+
+def _child_command_block() -> str:
+ """The argv the parent builds for each cycle."""
+ source = _smoke_source()
+ start = source.index("cmd = [\n sys.executable,")
+ end = source.index("proc = subprocess.run(cmd)", start)
+ return source[start:end]
+
+
+def test_every_option_the_child_needs_actually_reaches_the_child():
+ """The class of bug, not one instance of it.
+
+ Cycles run in fresh child processes and the parent rebuilds their argv from
+ an explicit list. A flag added to the parser but not to that list is
+ accepted on the command line, parsed, logged in the driver's exec line, and
+ silently ignored -- which is exactly what happened to --export-gguf on
+ kernel unsloth-probe-default-gguf-637565: the leg failed with "GGUF export
+ was never run" while the driver log showed --export-gguf right there in the
+ command.
+
+ --check-batched-generation escaped this only because it defaults to True, so
+ the child got it without being told. That is luck, not design.
+ """
+ import argparse
+ import importlib
+
+ sys.path.insert(0, str(SMOKE_DIR))
+ module = importlib.import_module("run_t4_smoke")
+
+ # Build the parser the same way main() does, by calling it with a sentinel
+ # that makes it return rather than run.
+ parser = argparse.ArgumentParser()
+ source = _smoke_source()
+ dests = set(re.findall(r'dest\s*=\s*"([a-z_0-9]+)"', source))
+ dests |= {
+ m.replace("-", "_") for m in re.findall(r'ap\.add_argument\(\s*"--([a-z0-9-]+)"', source)
+ }
+ assert "export_gguf" in dests, "the parser no longer defines --export-gguf"
+ assert "model" in dests, "the dest scrape found nothing; fix the scrape"
+
+ block = _child_command_block()
+ forwarded = {m.replace("-", "_") for m in re.findall(r'"--([a-z0-9-]+)"', block)}
+
+ missing = sorted(d for d in dests - forwarded - PARENT_ONLY_DESTS if not d.startswith("no_"))
+ assert not missing, (
+ f"these options are parsed but never forwarded to the cycle child, so "
+ f"setting them does nothing: {missing}. Either forward them or list "
+ f"them in PARENT_ONLY_DESTS with a reason."
+ )
+ del parser, module
+
+
+def test_the_export_flag_is_forwarded_as_a_bare_flag():
+ """store_true options cannot ride the (flag, value) loop -- `--export-gguf
+ True` is not a thing -- so they need their own append, and that is the line
+ that was missing."""
+ block = _child_command_block()
+ assert 'cmd.append("--export-gguf")' in block
+ assert "if args.export_gguf:" in block
+
+
+def test_the_export_settings_ride_the_value_loop():
+ block = _child_command_block()
+ assert '("--gguf-quantization", args.gguf_quantization)' in block
+ assert '("--gguf-accept", args.gguf_accept)' in block
+
+
+def test_a_second_cycle_cannot_report_an_already_installed_llama_cpp_as_a_source_build():
+ """Measured on unsloth-probe-visleg-full-b3a317, and it is a trap.
+
+ The prebuilt banner is printed once, by the install that downloads the
+ bundle. A second cycle in the same session finds llama.cpp already there
+ and prints nothing, so the field read `prebuilt: true` on cycle 0 and
+ `prebuilt: false` on cycle 1 for the SAME installation -- and `false` reads
+ as "built from source", which is the one thing this field exists to catch.
+
+ None is the third state: this run did not install it, so it cannot say.
+ """
+ from gguf_export import llama_cpp_facts
+
+ quiet = llama_cpp_facts("", ())
+ assert (
+ quiet["prebuilt"] is None
+ ), "an install that printed nothing must not claim a source build"
+ assert quiet["source_build_markers"] == []
+
+ # And the two real answers are unchanged.
+ assert (
+ llama_cpp_facts(
+ "Unsloth: Installing prebuilt llama.cpp b10472 - skipping compilation.", ()
+ )["prebuilt"]
+ is True
+ )
+ assert llama_cpp_facts("cmake --build . -j 4\n", ())["prebuilt"] is False
+
+
+def test_the_text_leg_gguf_export_does_not_land_in_the_artifact_directory():
+ """`/kaggle/working` is 21.0 GB and is what `kernels output` ships back.
+
+ Measured on unsloth-probe-lcleg-final-a90fbb, which is the run that found
+ this:
+
+ RuntimeError: Unsloth: Not enough disk space to convert to GGUF.
+ The export needs about 16.6GB on the filesystem holding
+ `/kaggle/working/t4_out_Latest_compile/cycle0/gguf_run0`
+
+ Two failures in one. A merge too big for that volume kills the leg, and a
+ merge that DOES fit is downloaded as part of the artifact, which nobody
+ wanted. gpt-oss and the vision run were both moved to a tempdir for exactly
+ this reason and this path was missed -- so a small model kept passing and
+ hid it.
+ """
+ src = _smoke_source()
+ call = src[src.index("gguf_export_record = export_gguf(") :]
+ call = call[: call.index(")")]
+ assert "tempfile.mkdtemp" in call, "the export writes into the artifact directory"
+ assert (
+ "args.outdir" not in call
+ ), "args.outdir is /kaggle/working, which is 21 GB and is collected"
diff --git a/tests/kaggle/test_t4_smoke_harness.py b/tests/kaggle/test_t4_smoke_harness.py
index d8c8fe3ac72..beea60737e8 100644
--- a/tests/kaggle/test_t4_smoke_harness.py
+++ b/tests/kaggle/test_t4_smoke_harness.py
@@ -1544,7 +1544,23 @@ def test_every_setting_the_child_needs_is_forwarded_to_it():
# ------------------------------------------------------------ kernel build
-LEG_NAMES = ("control", "canary", "gptoss", "grpo")
+LEG_NAMES = (
+ "control",
+ "canary",
+ "gptoss",
+ "grpo",
+ "default",
+ # Unpinned by design (all_cards), so its generated cells differ from every
+ # other leg's: the visible-GPU assertion in the preamble compares against 2.
+ "multi_gpu",
+ "latest_compile",
+ "vision_fla_compile",
+ # Retired in favour of vision_fla_compile and kept buildable: an unwired
+ # leg is unwired because of its install or an open question, not because
+ # the payload rots, and a retirement that stops building is one nobody can
+ # reverse in the hour they need it.
+ "frontier",
+)
def _build(
@@ -1740,7 +1756,7 @@ def test_every_leg_carries_the_version_recorder(tmp_path):
assert "versions.py" in COMMON_FILES
for name in LEGS:
- payload = _payload_notebooks(_build(tmp_path / name, name))[f"t4_{name}.ipynb"]
+ payload = _payload_notebooks(_build(tmp_path / name, name))[f"t4_{LEGS[name].name}.ipynb"]
assert "versions.py" in _cell(payload, 0)
assert "versions.flatten_versions" in _cell(payload, 2)
@@ -1757,7 +1773,7 @@ def test_every_registered_leg_is_either_carried_or_explicitly_unwired():
carried = [name for kernel in KERNELS for name in kernel]
assert len(carried) == len(set(carried)), carried
- assert sorted(carried) + sorted(UNWIRED) == sorted(LEGS), (
+ assert sorted(carried + list(UNWIRED)) == sorted(LEGS), (
sorted(carried),
sorted(UNWIRED),
sorted(LEGS),
@@ -1810,10 +1826,12 @@ def test_generated_cells_compile(tmp_path):
for index, cell in enumerate(nb["cells"]):
compile("".join(cell["source"]), f"{path}/{name}#cell{index}", "exec")
seen += 1
- # 5 builds x (3 driver cells + 4 payload cells), asserted so a refactor that
- # stops reaching the payloads cannot leave this test passing while compiling
- # nothing that matters.
- assert seen == 5 * 7, seen
+ # One build per leg in LEG_NAMES x (3 driver cells + 4 payload cells),
+ # asserted so a refactor that stops reaching the payloads cannot leave this
+ # test passing while compiling nothing that matters. Derived from LEG_NAMES
+ # rather than hardcoded: the literal 5 silently meant "every leg" only for
+ # as long as nobody added one.
+ assert seen == (len(LEG_NAMES) + 1) * 7, seen
def _undefined_names(source: str, already_bound: set) -> tuple:
@@ -2139,9 +2157,21 @@ def test_the_files_the_payload_carries_are_byte_identical_to_the_repo(tmp_path):
"training_evidence.py",
"run_t4_smoke.py",
"determinism.py",
+ "gguf_export.py",
+ "naive_trl_compare.py",
+ "kernel_provenance.py",
"pins/control.txt",
"references/t4_qwen2.5-0.5b.json",
}, sorted(files)
+ # Cross-checked against the registry as well as the literal above. The
+ # literal is what makes an accidental addition visible in a diff; the
+ # registry check is what stops the two drifting, which is how a leg ends up
+ # shipping a file no test knows about.
+ from legs import LEGS
+
+ leg = LEGS["control"]
+ declared = set(leg.files) | ({f"references/{leg.reference}"} if leg.reference else set())
+ assert set(files) == declared, sorted(declared)
for name, data in files.items():
assert gzip.decompress(base64.b64decode(data)) == (SMOKE_DIR / name).read_bytes(), name
@@ -2279,6 +2309,99 @@ def test_the_grpo_leg_names_its_attention_backend():
assert LEGS["grpo"].env.get("VLLM_ATTENTION_BACKEND") == "TRITON_ATTN"
+def test_the_grpo_leg_disables_flashinfer_at_the_only_layer_that_holds():
+ """Two Kaggle probes died four rungs each on the same flashinfer link, one
+ of them WITH `VLLM_USE_FLASHINFER_SAMPLER=0` already set, because that is
+ not the layer the decision is made at. unsloth_zoo's patch_vllm assigns
+ both vLLM env vars itself during model load:
+
+ vllm_utils.py:2494 VLLM_ATTENTION_BACKEND = "FLASHINFER"
+ vllm_utils.py:2502 VLLM_USE_FLASHINFER_SAMPLER = "1"
+
+ each behind `elif Version(vllm_version) >= Version("0.11.0")`, which this
+ leg's 0.19.1 pin satisfies. Its guard checks only that nvcc and ninja are
+ present, and on Kaggle both are - the missing piece is the driver stub the
+ compiled objects link against, which that check does not look at. So the
+ two settings above are overwritten before vLLM ever reads them.
+
+ UNSLOTH_VLLM_NO_FLASHINFER is read at vllm_utils.py:2449, ahead of every
+ assignment, and is the only one of the three that survives. Losing it puts
+ the leg straight back on a link that cannot succeed on this image."""
+ from legs import LEGS
+ assert LEGS["grpo"].env.get("UNSLOTH_VLLM_NO_FLASHINFER") == "1"
+
+
+def test_the_grpo_leg_removes_flashinfer_and_the_removal_reaches_the_payload(tmp_path):
+ """The env vars were necessary and NOT sufficient, established over four
+ Kaggle ladder probes:
+
+ r1 nothing set 4/4 rungs fail, cached_ops/sampling
+ r2 VLLM_USE_FLASHINFER_SAMPLER=0 identical failure
+ r3 UNSLOTH_VLLM_NO_FLASHINFER=1 sampling build gone; now fails in
+ cached_ops/batch_prefill_with_kv_cache_...
+ r4 uninstall flashinfer PASSES on the first rung, 11.30 GB
+
+ r3 is the one that proves the remaining gap: the sampler build genuinely
+ stopped, and the failure moved to ATTENTION, which is chosen by
+ VLLM_ATTENTION_BACKEND -- already TRITON_ATTN and confirmed inherited by the
+ rung child -- and which vLLM offers no prefill-specific opt-out for. There
+ was no variable left to set, which is why the package goes instead.
+
+ THE SHIPPED LEG FAILS DIFFERENTLY, and that is the reason this guard is
+ worth keeping rather than a historical note. Removing the uninstall from the
+ leg (unsloth-probe-grpo-noun-05777b, nothing else changed) does NOT
+ reproduce the link error at all -- the leg carries a libcuda shim
+ (run_grpo_t4.py:597, /usr/local/cuda/compat/libcuda.so, since #8440) that
+ the probes lacked. It fails instead with
+
+ AcceleratorError: CUDA error: an illegal memory access was encountered
+
+ which is verbatim the intermittent crash UNWIRED["grpo"] documents from
+ kernels unsloth-t4-ci-70a2f4eb and -c98f14be. With the uninstall the crash
+ has not recurred.
+
+ State the strength honestly: that crash is documented at roughly two failures
+ in three, so a handful of clean runs is suggestive and not proof, and this
+ guard asserts only that the uninstall is PRESENT and REACHES the payload --
+ which removing it demonstrably breaks. It does not claim to have identified
+ the crash's cause.
+
+ Asserted through the BUILT payload rather than off the dataclass: a field
+ nothing emits is a setting that reads like coverage and does nothing, and
+ that is the exact failure mode this file exists to catch.
+ """
+ from legs import LEGS
+
+ assert set(LEGS["grpo"].uninstall) >= {"flashinfer-python", "flashinfer-cubin"}
+ payload = _payload_notebooks(_build(tmp_path / "g", "grpo"))["t4_grpo.ipynb"]
+ install_cell = _cell(payload, 1)
+ assert "flashinfer-python" in install_cell
+ assert "uninstall" in install_cell
+
+
+def test_a_leg_with_nothing_to_uninstall_does_not_run_pip_uninstall(tmp_path):
+ """The counterpart, so the emission is conditional rather than always-on:
+ an unconditional `pip uninstall -y` with an empty list is a per-leg
+ subprocess that can only cost time."""
+ from legs import LEGS
+
+ assert LEGS["control"].uninstall == ()
+ payload = _payload_notebooks(_build(tmp_path / "c", "control"))["t4_control.ipynb"]
+ assert "flashinfer" not in _cell(payload, 1)
+
+
+def test_the_grpo_leg_asks_for_the_utilization_both_platforms_measured(tmp_path):
+ """0.95 is measured, not requested-and-hoped. Colab (torch 2.11.0) peaked at
+ 11.76 GB and Kaggle (torch 2.10.0) at 11.30 GB, both on the FIRST rung of a
+ 0.95/0.8/0.6/0.5 ladder and both surviving three sleep/wake cycles, roughly
+ 3 GB under a 14.56 GB card. The 0.5 that used to be here came from Qwen3-4B
+ probes and was never measured for the 0.6B this leg trains."""
+ from legs import LEGS
+
+ args = LEGS["grpo"].args
+ assert args[args.index("--gpu-memory-utilization") + 1] == "0.95"
+
+
def test_the_grpo_leg_no_longer_carries_xformers():
"""Its vLLM backend is gone at this version, so it would be a package
nothing selects, resolved against a torch it has opinions about."""
@@ -2305,13 +2428,74 @@ def test_nothing_is_both_wired_and_unwired():
assert not (wired & set(UNWIRED)), sorted(wired & set(UNWIRED))
-def test_an_unwired_note_says_what_is_unknown():
+def test_an_unwired_note_says_what_is_unknown_or_what_replaced_it():
"""An unwired leg whose note reads as settled is a leg someone wires without
- running it. Vacuous while UNWIRED is empty, and correctly so: it has
- something to check the moment a leg goes back in."""
- from legs import UNWIRED
+ running it.
+
+ THREE reasons a leg is not wired, and they are not the same thing.
+
+ A leg with an open question says STILL UNKNOWN and names it.
+
+ A leg that was REPLACED has nothing unknown about it at all -- frontier is
+ retired because vision_fla_compile asserts everything it did and more --
+ and forcing that note to claim an open question would be a lie in the file
+ that exists to stop lies of exactly that kind.
+
+ A leg that was MEASURED AND REJECTED is the third, added when multi_gpu hit
+ it: the leg passes on hardware and was still held out, because an A/B
+ showed it costing wall clock and breaking another leg. Nothing about it is
+ unknown and nothing replaced it, so the first two categories would both be
+ false. What that note owes the reader instead is the way back in, so it
+ must say WHAT WOULD UNBLOCK IT -- otherwise "rejected" reads as permanent
+ and the measurement behind it is never revisited.
+
+ A leg that is UNDER RE-MEASUREMENT is the fourth, and it exists because a
+ rejection can turn out to rest on a defect somewhere else. multi_gpu was
+ rejected partly for breaking the Default leg; that break was this driver
+ building the leg on the wrong python and installing pyarrow where dill
+ pickles it by value, and it is fixed. Leaving the note reading REJECTED
+ would keep a withdrawn measurement standing as a finding, which is the
+ failure this file exists to catch, and forcing it to read STILL UNKNOWN
+ would throw away everything that IS known. Such a note owes the reader the
+ run that will settle it, by name, so the answer is collectable rather than
+ perpetually pending.
+
+ So the rule is: say which of the four it is, name the superseding leg if it
+ was superseded, name the unblock condition if it was rejected, and name the
+ deciding run if it is being re-measured. A bare note passes none of them.
+ """
+ from legs import LEGS, UNWIRED
for name, note in UNWIRED.items():
- assert "STILL UNKNOWN" in note, f"{name} note does not say what is open"
+ if "SUPERSEDED" in note:
+ named = [other for other in LEGS if other != name and other in note]
+ assert named, (
+ f"{name} says it was superseded and does not name what by; a "
+ f"retirement nobody can trace is a deletion with extra steps"
+ )
+ continue
+ if "MEASURED AND REJECTED" in note:
+ assert "WHAT WOULD UNBLOCK IT" in note, (
+ f"{name} says it was measured and rejected without saying what "
+ f"would change that; a rejection with no way back is a deletion "
+ f"that keeps costing a reader the time to re-derive it"
+ )
+ continue
+ if "UNDER RE-MEASUREMENT" in note:
+ assert "WHAT WOULD UNBLOCK IT" in note, (
+ f"{name} says it is being re-measured without saying what would "
+ f"settle it, which is a rejection with the deadline removed"
+ )
+ # The deciding run has to be NAMED. "we will re-run it" is how a
+ # measurement stays pending for a year.
+ assert re.search(r"\b(ab\d+|unsloth-probe-[a-z0-9-]+)\b", note), (
+ f"{name} is under re-measurement and names no run that will "
+ f"decide it, so nobody can go and read the answer"
+ )
+ continue
+ assert "STILL UNKNOWN" in note, (
+ f"{name} note says none of: what is open, what replaced it, or "
+ f"what would unblock it"
+ )
def test_grpo_stays_unwired_while_the_illegal_memory_access_is_open():
@@ -2359,12 +2543,21 @@ def test_the_grpo_leg_keeps_the_config_that_actually_fit():
unsloth_zoo/gradient_checkpointing.py:1013, peaking at 15.97GB in 16-bit and
19.25GB in 4-bit. The set below passed on kernel unsloth-t4-ci-53efcc4e at
13.60GB with reward_std 0.707 at step 2, so restoring any of them to the
- notebook's value is a session that OOMs, and it fails here instead."""
+ notebook's value is a session that OOMs, and it fails here instead.
+
+ UTILIZATION IS THE ONE EXCEPTION, raised 0.5 -> 0.95 on measurement. Those
+ probes trained **Qwen3-4B**; this leg trains Qwen3-0.6B, where the
+ activation budget is a different problem entirely. A 0.95/0.8/0.6/0.5 ladder
+ stopped at the FIRST rung on both platforms - Colab peak reserved 11.76GB,
+ Kaggle 11.30GB, three sleep/wake cycles each - roughly 3GB under the card.
+ The other three values are unchanged and still carry the 4B evidence, which
+ is why they are asserted together with this one rather than relaxed with
+ it."""
from legs import LEGS
args = LEGS["grpo"].args
for flag, value in (
- ("--gpu-memory-utilization", "0.5"),
+ ("--gpu-memory-utilization", "0.95"),
("--max-seq-length", "1024"),
("--num-generations", "2"),
("--lora-rank", "16"),
@@ -3097,6 +3290,12 @@ def test_an_alternate_spelling_of_the_step_count_keeps_the_reference_band(tmp_pa
source, _, name = inner.rpartition(".")
if source == "steps.stepcount.outputs":
return validated.get(name, "")
+ # A dispatch-only or event-only expression. On the pull_request event
+ # this test models, `inputs` is null and the schedule branch is false,
+ # so GitHub substitutes the empty string -- which is what the leg-list
+ # override reads as, and is the shape that must keep --all-kernels.
+ if inner.startswith("inputs.") or inner.startswith("github."):
+ return ""
raise AssertionError(
f"the build step takes {key} from {expr}. The step count it builds with, "
f"and the reference count it compares against, have to be the ones the "
@@ -3236,11 +3435,33 @@ def test_the_workflow_is_never_preempted_by_the_capacity_sweeper():
# ----------------------------------------------------------------- report
+_TWO_LEGS = [
+ {"label": "control", "passed": True, "steps": []},
+ {"label": "canary", "passed": True, "steps": []},
+]
+_A_FAILING_LEG = [
+ {"label": "control", "passed": False, "failures": ["x"], "steps": []},
+ {"label": "canary", "passed": True, "steps": []},
+]
+# The Studio payload from the merged kernel. Not this reporter's.
+_STUDIO = [{"label": "studio-gpu", "passed": False, "failures": ["x"], "assertions": []}]
+
+
@pytest.mark.parametrize(
- ("verdict", "expected_exit"),
- [("pass", 0), ("partial", 0), ("infra", 0), ("fail", 1)],
+ ("verdict", "reports", "expected_exit"),
+ [
+ ("pass", _TWO_LEGS, 0),
+ ("partial", [], 0),
+ ("infra", [], 0),
+ ("fail", _A_FAILING_LEG, 1),
+ # Since --with-studio the kernel verdict covers a payload this reporter
+ # does not own, so a failing Studio run must not print "Kaggle T4
+ # smoke: FAIL" over two green legs. kaggle_studio_ci/report.py turns
+ # that red, in the same job, from the same evidence directory.
+ ("fail", _TWO_LEGS + _STUDIO, 0),
+ ],
)
-def test_only_a_real_assertion_failure_turns_the_job_red(tmp_path, verdict, expected_exit):
+def test_only_a_real_assertion_failure_turns_the_job_red(tmp_path, verdict, reports, expected_exit):
evidence = tmp_path / "evidence"
evidence.mkdir()
(evidence / "launch_result.json").write_text(
@@ -3250,7 +3471,7 @@ def test_only_a_real_assertion_failure_turns_the_job_red(tmp_path, verdict, expe
"reason": "test",
"slug": "u/s",
"kernel_state": "COMPLETE",
- "reports": [],
+ "reports": reports,
}
)
)
@@ -4258,36 +4479,41 @@ def test_the_frontier_leg_does_not_carry_the_zoo_requirement():
)
-def test_the_frontier_leg_rides_in_the_one_kernel_rather_than_opening_a_session():
- """frontier must never be the reason a second Kaggle session is opened.
-
- This used to read "the seat that costs nothing", on the basis that a
- session bills its wall clock once rather than per card, so the second
- kernel's idle T4 carried frontier for free. That is still true about
- BILLING and is no longer the binding constraint. The account allows two
- concurrent GPU sessions; two kernels took both, and
- kaggle-t4-studio-gpu-ci.yml runs on the same account, so the notebook leg
- locked Unsloth out for as long as it ran. Everything now packs into one
- kernel and frontier queues behind the legs ahead of it.
+def test_frontier_is_retired_in_favour_of_the_leg_that_supersedes_it():
+ """frontier no longer runs, and this used to assert that it did.
- So the property worth pinning is no longer "frontier shares a full kernel"
- -- that assertion passes just as happily on a two-session layout -- but
- "there is one kernel and frontier is in it".
+ The property it pinned was "there is one kernel and frontier is in it",
+ which mattered when frontier was the cheapest way to cover a
+ latest-everything stack. `vision_fla_compile` covers that stack on a bigger
+ model AND asserts the vendored FLA kernels, the Turing attention choice, a
+ real vision training run, the merged vision export, a Q8_0 GGUF with its
+ mmproj sidecar and inference on the exported file. Everything frontier
+ proved is a subset.
- It has passed on real hardware in both layouts: transformers 5.15.0 /
- trl 1.9.2 paired, and transformers 5.15.1 / trl 1.10.0 on run 32607621452,
- ten steps, canary emitted, two fresh processes agreeing bitwise.
+ Rewritten rather than deleted, because the thing worth guarding now is that
+ the retirement was DELIBERATE: a leg that quietly falls out of KERNELS with
+ no entry anywhere is indistinguishable from one dropped by a bad merge.
"""
- sys.path.insert(0, str(CI_DIR))
- import legs
+ from legs import KERNELS, UNWIRED
- wired = [kernel for kernel in legs.KERNELS if "frontier" in kernel]
- assert len(wired) == 1, f"frontier appears in {len(wired)} kernels, expected 1"
- assert len(legs.KERNELS) == 1, (
- f"{len(legs.KERNELS)} kernels means {len(legs.KERNELS)} concurrent Kaggle "
- "sessions, and the account only has two. Unsloth needs one of them."
+ assert "frontier" not in {name for kernel in KERNELS for name in kernel}, (
+ "frontier is back in the wired set; if that is intended, this test and "
+ "its UNWIRED note both need rewriting rather than deleting"
)
- assert "frontier" not in legs.UNWIRED
+ assert "frontier" in UNWIRED, (
+ "frontier is in neither KERNELS nor UNWIRED, which is what a leg "
+ "dropped by accident looks like"
+ )
+ assert (
+ "SUPERSEDED" in UNWIRED["frontier"]
+ ), "the note must say it was replaced, not that it is broken"
+ assert "vision_fla_compile" in UNWIRED["frontier"], "and by what"
+ # And the leg it names is really carrying the load.
+ assert "vision_fla_compile" in {name for kernel in KERNELS for name in kernel}
+
+ # There is still exactly ONE kernel, which is the half of the old
+ # assertion that was never about frontier.
+ assert len(KERNELS) == 1, KERNELS
def test_the_pinned_kaggle_client_carries_the_calls_this_workflow_makes():
diff --git a/tests/kaggle/test_venv_interpreter_and_overlay_path.py b/tests/kaggle/test_venv_interpreter_and_overlay_path.py
new file mode 100644
index 00000000000..0500f4eda2f
--- /dev/null
+++ b/tests/kaggle/test_venv_interpreter_and_overlay_path.py
@@ -0,0 +1,197 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""Two defects that presented as one, and neither names this driver in its
+traceback.
+
+Measured over six sessions from ONE commit, deterministic per arm:
+
+| arm | Default leg python | torch | datasets | verdict |
+|---|---|---|---|---|
+| A (3 runs) | 3.12.13 | 2.10.0+cu128 | 4.3.0 | pass 3/3 |
+| B (3 runs) | **3.13.13** | 2.12.1+cu130 | 5.0.1 | **fail 3/3** |
+
+Every other leg in both arms stayed on 3.12.13.
+
+**Defect one: the venv interpreter was never pinned.** uv's default
+python-preference is `managed`, so once any managed CPython exists under
+``~/.local/share/uv/python`` a bare ``uv venv`` builds on that instead of the
+Kaggle image's interpreter. ``--system-site-packages`` is still accepted and
+still inherits nothing, because a 3.13 venv cannot see a 3.12 site-packages. The
+leg then resolves torch, datasets and pyarrow from PyPI -- a stack no Kaggle
+user has -- and pays minutes for the privilege, with nothing red.
+
+**Defect two: the overlay directory was hostile to dill**, which is what turned
+the wrong interpreter into a crash. ``dill._dill._is_builtin_module`` pickles a
+module by REFERENCE only if its ``__file__`` starts with a sys prefix, ends with
+an extension suffix, or contains the literal string ``site-packages``. A plain
+``pip install --target /tmp/t4ci_venvs/overlay_X`` satisfies none of the three,
+so everything in the overlay is pickled BY VALUE. ``Dataset.from_dict``
+fingerprints through dill, ``datasets/utils/_dill.py:_save_arrowTable`` saves
+``create_arrowTable``, dill walks its globals into the pyarrow module and dies
+on pyarrow's Cython ``MonthDayNano``, whose ``__module__`` is ``builtins``:
+
+ PicklingError: Can't pickle <class 'MonthDayNano'>:
+ it's not found as builtins.MonthDayNano
+
+Reproduced on CPU in seconds against a byte-identical package tree with the
+DIRECTORY NAME as the only variable: the plain ``--target`` directory raised,
+the copy named ``site-packages`` returned a fingerprint.
+
+The rules below are asserted through the GENERATED driver source and, for the
+dill half, by executing the real path expression against the real dill
+predicate. A rule written against the template would not have caught either.
+"""
+
+from __future__ import annotations
+
+import pathlib
+import re
+import sys
+import types
+
+import pytest
+
+ROOT = pathlib.Path(__file__).resolve().parents[2]
+SMOKE_DIR = ROOT / "tests" / "kaggle" / "t4_smoke"
+sys.path.insert(0, str(ROOT / ".github" / "scripts"))
+
+from kaggle_t4_ci import build_kernel, legs # noqa: E402
+
+
+def _driver_source() -> str:
+ """The driver as the kernel will actually run it, overlays included."""
+ names = ("default", "canary")
+ payloads = {
+ f"t4_{legs.LEGS[n].name}.ipynb": build_kernel.build_payload_notebook(
+ SMOKE_DIR, legs.LEGS[n], unsloth_ref = "main", zoo_ref = "main"
+ )
+ for n in names
+ }
+ driver = build_kernel.build_driver(
+ payloads,
+ per_run_timeout = 3600,
+ vram_source = {f"t4_{legs.LEGS[n].name}.ipynb": legs.LEGS[n] for n in names},
+ overlays = {f"t4_{legs.LEGS[n].name}.ipynb": tuple(legs.LEGS[n].overlay) for n in names},
+ )
+ return "".join("".join(c["source"]) for c in driver["cells"])
+
+
+# --------------------------------------------------------------------------
+# Defect one: the interpreter
+# --------------------------------------------------------------------------
+
+
+def test_the_venv_is_built_on_the_drivers_own_interpreter():
+ """`--system-site-packages` on a different python version is a silent
+ no-op, so the flag alone is not the guard. The interpreter is."""
+ source = _driver_source()
+ body = source.split("def _make_venv(")[1].split("def run_one(")[0]
+ cmd = re.search(r"venv_cmd = \[([^\]]*)\]", body)
+ assert cmd, "the uv venv command is no longer a literal list; re-read this rule"
+ assert '"--python"' in cmd.group(1) and "sys.executable" in cmd.group(1), (
+ "uv venv runs without --python, so uv's `managed` python-preference "
+ "decides the version. That is how the Default leg came up on 3.13 "
+ "against a 3.12 image and inherited no system site-packages at all: "
+ f"{cmd.group(1)}"
+ )
+
+
+def test_the_system_site_flag_is_still_passed():
+ """The pin does not replace it. Both are needed: the right interpreter with
+ no inheritance resolves the whole stack from PyPI just as expensively."""
+ body = _driver_source().split("def _make_venv(")[1].split("def run_one(")[0]
+ assert '"--system-site-packages"' in body
+
+
+def test_the_resulting_python_version_is_REPORTED():
+ """Three sessions went red before anyone read a python version out of a
+ leg. A venv on the wrong interpreter is invisible without this."""
+ body = _driver_source().split("def _make_venv(")[1].split("def run_one(")[0]
+ sentinel = body.split("_VENV ")[1]
+ assert '"python"' in sentinel, (
+ "the _VENV record does not carry the venv's python version, so the "
+ "next wrong-interpreter run is again only visible by inference"
+ )
+ assert (
+ "sys.version_info" in body
+ ), "nothing compares the venv against the driver's own interpreter"
+
+
+# --------------------------------------------------------------------------
+# Defect two: the overlay directory, driven against the real dill predicate
+# --------------------------------------------------------------------------
+
+
+def _overlay_dir_from_generated_source() -> pathlib.Path:
+ """Evaluate the driver's OWN `_ov_dir` expression.
+
+ Not a hand-written copy of the path: the rule below is only worth having if
+ it fails when the generated expression changes.
+ """
+ source = _driver_source()
+ line = [ln.strip() for ln in source.splitlines() if ln.strip().startswith("_ov_dir = ")]
+ assert len(line) == 1, f"expected one _ov_dir assignment, found {len(line)}"
+ scope = {
+ "VENV_ROOT": pathlib.Path("/tmp/t4ci_venvs"),
+ "pathlib": pathlib,
+ "name": "t4_Default.ipynb",
+ }
+ exec(line[0], scope) # noqa: S102 - the driver's own line, by design
+ return scope["_ov_dir"]
+
+
+def test_the_overlay_lands_where_dill_will_pickle_by_reference():
+ """The real dill predicate, against the real generated path.
+
+ `_is_builtin_module` needs only `__file__`, so a module object standing in
+ for pyarrow is enough to execute the decision that crashed the leg.
+ """
+ dill_dill = pytest.importorskip("dill._dill")
+ ov = _overlay_dir_from_generated_source()
+
+ stand_in = types.ModuleType("pyarrow")
+ stand_in.__file__ = str(ov / "pyarrow" / "__init__.py")
+ assert dill_dill._is_builtin_module(stand_in), (
+ f"a package installed at {ov} is pickled BY VALUE by dill, so any leg "
+ "whose overlay carries pyarrow dies in Dataset.from_dict with "
+ "PicklingError: Can't pickle <class 'MonthDayNano'>"
+ )
+
+
+def test_the_mutation_this_rule_exists_for_is_actually_caught():
+ """The negative control. Without it the rule above could be passing because
+ dill answers True for everything, which it does not."""
+ dill_dill = pytest.importorskip("dill._dill")
+ before = types.ModuleType("pyarrow")
+ # The path the driver used until this was found: a --target directory that
+ # is neither under a sys prefix nor named site-packages.
+ before.__file__ = "/tmp/t4ci_venvs/overlay_t4_Default/pyarrow/__init__.py"
+ assert not dill_dill._is_builtin_module(before), (
+ "dill no longer distinguishes the two paths, so the fix above may have "
+ "become unnecessary -- re-measure before simplifying it away"
+ )
+
+
+def test_the_path_is_not_merely_a_string_containing_site_packages():
+ """`site-packages` has to be the directory packages are installed INTO, not
+ a decoration further up the path, or pip lays the tree somewhere dill still
+ reads by value."""
+ ov = _overlay_dir_from_generated_source()
+ assert ov.name == "site-packages", (
+ f"the overlay --target is {ov}, whose last component is {ov.name!r}; "
+ "dill's check is on the installed module's own __file__"
+ )
+
+
+def test_the_overlay_target_flag_uses_that_directory():
+ """The path can be right and unused. This is the link between them."""
+ source = _driver_source()
+ install = source.split('"--target", str(_ov_dir)')
+ assert len(install) == 2, (
+ "the overlay install no longer targets _ov_dir, so the directory the "
+ "rules above check is not where packages land"
+ )
+ assert (
+ 'env["PYTHONPATH"] = str(_ov_dir)' in source
+ ), "the overlay is installed somewhere the payload never reads"
diff --git a/tests/kaggle/test_vision_run.py b/tests/kaggle/test_vision_run.py
new file mode 100644
index 00000000000..1ddd037635a
--- /dev/null
+++ b/tests/kaggle/test_vision_run.py
@@ -0,0 +1,344 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
+
+"""The vision run, and the one way it goes green while testing nothing.
+
+A "vision run" that never puts an image on the GPU is a text run in a costume.
+It trains, its loss falls, its adapter updates, and every assertion a text leg
+makes passes. TRL will produce exactly that state if
+`remove_unused_columns=False` is dropped, because the image column is removed
+before the collator ever sees it.
+
+So the rules are read off a REAL collated batch, and the guards below are
+calibrated to catch the costume rather than the crash.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+PAYLOAD = ROOT / "tests" / "kaggle" / "t4_smoke"
+sys.path.insert(0, str(PAYLOAD))
+
+from run_vision_t4 import build_conversations, vision_failures # noqa: E402
+
+SRC = (PAYLOAD / "run_vision_t4.py").read_text(encoding = "utf-8")
+
+
+def _args(**over):
+ base = dict(
+ max_steps = 2,
+ require_vision_lora = True,
+ export = False,
+ )
+ base.update(over)
+ return argparse.Namespace(**base)
+
+
+def _good(**over):
+ result = {
+ "pixels": {
+ "columns": ["input_ids", "labels", "pixel_values"],
+ "pixel_keys": ["pixel_values"],
+ "pixel_sizes": {"pixel_values": {"numel": 602112, "shape": [1, 3, 448, 448]}},
+ },
+ "vision_lora": {"vision_module_count": 48, "language_modules": 112},
+ "metrics": [{"step": 1, "loss": 3.0}, {"step": 2, "loss": 2.0}],
+ "adapter_update": {"before": 0.0, "after": 91.2, "tensors": 864, "changed": True},
+ "generated": "\\frac{1}{2}",
+ }
+ result.update(over)
+ return result
+
+
+def test_the_measured_shape_passes():
+ assert vision_failures(_good(), _args()) == []
+
+
+def test_a_batch_with_no_pixels_is_the_headline_failure():
+ """The costume. Everything else about this run is healthy."""
+ broken = vision_failures(
+ _good(pixels = {"columns": ["input_ids", "labels"], "pixel_keys": [], "pixel_sizes": {}}),
+ _args(),
+ )
+ assert broken and "no pixel tensor at all" in broken[0]
+
+
+def test_an_empty_pixel_tensor_is_a_failure():
+ broken = vision_failures(
+ _good(pixels = {"pixel_sizes": {"pixel_values": {"numel": 0, "shape": [0]}}}),
+ _args(),
+ )
+ assert broken and "pixel tensors are empty" in broken[0]
+
+
+def test_a_lora_that_never_reached_the_vision_tower_is_a_failure():
+ """`finetune_vision_layers=True` is a request, not a result."""
+ broken = vision_failures(
+ _good(vision_lora = {"vision_module_count": 0, "language_modules": 112}), _args()
+ )
+ assert broken and "no LoRA module landed" in broken[0]
+
+
+def test_the_vision_lora_rule_can_be_turned_off_for_a_language_only_run():
+ assert (
+ vision_failures(
+ _good(vision_lora = {"vision_module_count": 0}), _args(require_vision_lora = False)
+ )
+ == []
+ )
+
+
+def test_an_adapter_that_did_not_move_is_a_failure():
+ """Starts at exactly zero by construction, so any movement is a real
+ optimizer step rather than a tolerance question."""
+ broken = vision_failures(
+ _good(adapter_update = {"before": 0.0, "after": 0.0, "tensors": 864, "changed": False}),
+ _args(),
+ )
+ assert broken and "did not move" in broken[0]
+
+
+def test_empty_or_missing_generation_is_a_failure():
+ assert vision_failures(_good(generated = ""), _args())
+ assert vision_failures(_good(generated = None), _args())
+
+
+def test_a_short_step_count_is_a_failure():
+ broken = vision_failures(_good(metrics = [{"step": 1, "loss": 3.0}]), _args())
+ assert broken and "expected 2 logged steps" in broken[0]
+
+
+def test_a_non_finite_loss_is_a_failure():
+ broken = vision_failures(
+ _good(metrics = [{"step": 1, "loss": float("nan")}, {"step": 2, "loss": 2.0}]), _args()
+ )
+ assert broken and "non-finite" in broken[0]
+
+
+def test_the_export_rule_only_fires_when_the_export_was_requested():
+ assert vision_failures(_good(), _args(export = False)) == []
+ broken = vision_failures(_good(), _args(export = True))
+ assert broken and "failed" in broken[0]
+ assert (
+ vision_failures(
+ _good(export = {"ok": True, "files": [{"name": "model.safetensors", "mb": 1200.0}]}),
+ _args(export = True),
+ )
+ == []
+ )
+
+
+def test_an_export_that_reported_ok_and_wrote_nothing_is_a_failure():
+ """The gpt-oss lesson: an export can succeed and leave no file anywhere."""
+ broken = vision_failures(
+ _good(export = {"ok": True, "files": [], "dir": "/tmp/x"}), _args(export = True)
+ )
+ assert broken and "wrote nothing" in broken[0]
+
+
+def test_the_conversation_shape_matches_the_notebook():
+ """Built from the notebook's own convert_to_conversation. A different shape
+ trains something no notebook produces, which tests the leg."""
+ rows = build_conversations([{"image": "IMG", "text": "x^2"}])
+ assert len(rows) == 1
+ messages = rows[0]["messages"]
+ assert messages[0]["role"] == "user"
+ kinds = [part["type"] for part in messages[0]["content"]]
+ assert "image" in kinds and "text" in kinds
+ assert messages[1]["role"] == "assistant"
+ assert messages[1]["content"][0]["text"] == "x^2"
+
+
+def test_the_four_settings_vision_training_needs_are_all_present():
+ """Dropping `remove_unused_columns = False` is the exact edit that turns
+ this into a text run: TRL removes the image column before the collator sees
+ it, and nothing raises."""
+ assert "remove_unused_columns = False" in SRC
+ assert 'dataset_text_field = ""' in SRC
+ assert 'dataset_kwargs = {"skip_prepare_dataset": True}' in SRC
+ assert "UnslothVisionDataCollator(model, tokenizer)" in SRC
+
+
+def test_the_pixel_evidence_is_read_before_training():
+ """After `trainer.train()` the dataloader has been consumed, and a
+ re-created one is not necessarily the object the trainer used."""
+ pixels_at = SRC.index('result["pixels"] = pixel_evidence(trainer)')
+ train_at = SRC.index("stats = trainer.train()")
+ assert pixels_at < train_at
+
+
+def test_the_export_does_not_land_in_the_artifact_volume():
+ """/kaggle/working is 21GB and a merged 2B is a meaningful fraction of it."""
+ assert 'tempfile.mkdtemp(prefix = "vision_export_")' in SRC
+ # Scoped to the export BLOCK. A whole-file search matches args.outdir in
+ # main(), where it is correct, and the assertion would fail for a reason
+ # that has nothing to do with the export.
+ # Anchored on the mkdtemp rather than on `if args.export:`, because that
+ # string appears FIRST in vision_failures() and the naive split lands in
+ # the wrong function -- which is how this assertion failed the first time.
+ block = SRC.split("export_dir = tempfile.mkdtemp", 1)[1].split('result["export"] = record', 1)[
+ 0
+ ]
+ assert "args.outdir" not in block, "the merged model is written into the artifact volume"
+ assert "model.save_pretrained_merged(export_dir, tokenizer)" in block
+
+
+def test_the_train_dataset_is_a_dataset_and_its_images_stay_pil():
+ """Two failures in one, both measured rather than guessed.
+
+ TRL 1.x rejects a plain list, which is what the notebook passes:
+
+ TypeError: `train_dataset` must be a `Dataset` or `IterableDataset`,
+ got `list`
+
+ And the obvious fix corrupts the data. `Dataset.from_list` Arrow-encodes a
+ nested PIL object into a `{bytes, path}` DICT on the way back out, so the
+ collator receives something that is not an image and nothing says so.
+
+ `with_transform` applies at access time, keeps the column's Image feature,
+ and still satisfies TRL's type check.
+ """
+ from datasets import Dataset as HFDataset
+ from PIL import Image
+
+ from run_vision_t4 import conversation_dataset
+
+ base = HFDataset.from_dict({"image": [Image.new("RGB", (8, 8))], "text": ["x^2"]})
+ built = conversation_dataset(base)
+ assert isinstance(built, HFDataset), "TRL 1.x rejects anything else"
+ image = built[0]["messages"][0]["content"][1]["image"]
+ assert isinstance(image, Image.Image), (
+ f"the image came back as {type(image).__name__}, which is the silent "
+ f"Arrow corruption Dataset.from_list produces"
+ )
+
+
+def test_from_list_would_have_corrupted_the_images():
+ """The negative control. Without it, the test above passes for a
+ `with_transform` that happens to work and says nothing about why the
+ obvious alternative was rejected."""
+ from datasets import Dataset as HFDataset
+ from PIL import Image
+
+ from run_vision_t4 import build_conversations
+
+ rows = build_conversations([{"image": Image.new("RGB", (8, 8)), "text": "x^2"}])
+ naive = HFDataset.from_list(rows)
+ image = naive[0]["messages"][0]["content"][1]["image"]
+ assert not isinstance(image, Image.Image), (
+ "from_list now preserves PIL, so the with_transform indirection may no "
+ "longer be needed; re-check before simplifying"
+ )
+
+
+def test_a_marker_that_matched_nothing_is_refused_rather_than_answered_no():
+ """Measured on `unsloth-probe-vision-train-r2-8ed253`, and it named the
+ wrong defect. PEFT calls these parameters `lora_B`, with a capital B; the
+ marker was matched against the raw name, so it matched none of the 864 of
+ them and summed to zero both before AND after. The run had trained
+ perfectly well -- loss 1.13 -> 0.56, a merged 4.3 GB export -- and the
+ report said the optimizer applied nothing.
+
+ Zero over zero tensors and zero over 864 tensors are opposite findings and
+ read identically, which is why the count is carried.
+ """
+ broken = vision_failures(
+ _good(adapter_update = {"before": 0.0, "after": 0.0, "tensors": 0, "changed": False}),
+ _args(),
+ )
+ assert broken, "a question that was never asked must not pass"
+ assert "never asked" in broken[0], broken
+ assert "did not move" not in broken[0], (
+ "reporting an unmatched marker as an untrained adapter sends the "
+ "reader after the wrong bug, which is what happened on hardware"
+ )
+
+
+def test_adapter_sum_finds_the_capital_b_peft_names():
+ """Drives the REAL function, because every rule above is fed a dict written
+ by hand and none of them execute the code that produces it. That is exactly
+ how the capital-B bug reached hardware."""
+ import torch
+
+ from run_vision_t4 import adapter_sum
+
+ class _Stub:
+ def named_parameters(self):
+ # The names PEFT actually emits, capitals and all.
+ yield (
+ "base_model.model.visual.blocks.0.attn.qkv.lora_A.default.weight",
+ torch.ones(2, 2),
+ )
+ yield (
+ "base_model.model.visual.blocks.0.attn.qkv.lora_B.default.weight",
+ torch.full((2, 2), 3.0),
+ )
+ yield (
+ "base_model.model.layers.0.self_attn.q_proj.lora_B.default.weight",
+ torch.full((2, 2), 1.0),
+ )
+
+ got = adapter_sum(_Stub())
+ assert got["tensors"] == 2, got
+ assert got["sum"] == 16.0, got
+
+
+def test_the_leg_actually_DRIVES_the_vision_run():
+ """The gap this closes was live for two rounds: `run_vision_t4.py` was in
+ the leg's `files` and nothing ever executed it, so Vision_FLA_compile
+ trained TEXT, asserted kernels, and shipped a payload it never ran.
+
+ A file that is copied and not run is the quietest kind of coverage there
+ is: every guard in this module passed, on a leg where the image path was
+ dead.
+ """
+ import sys as _sys
+
+ _sys.path.insert(0, str(ROOT / ".github" / "scripts" / "kaggle_t4_ci"))
+ import legs # noqa: E402
+
+ leg = legs.LEGS["vision_fla_compile"]
+ assert "--vision-run" in leg.args, "the leg ships the payload but never runs it"
+ assert "run_vision_t4.py" in leg.files
+ assert (
+ "--export-gguf" in leg.args
+ ), "the merged vision export is the half the text path cannot exercise"
+
+
+def test_the_parent_spawns_the_vision_run_after_the_cycles():
+ """Two 4bit models resident at once on a 14.56GB card is how a leg becomes
+ an OOM blamed on the thing it was testing. It is also what keeps a vision
+ failure from reading as a text-training one."""
+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")
+ assert '"run_vision_t4.py"' in src
+ cycles_at = src.index("runs.append(json.loads(report_file.read_text")
+ spawn_at = src.index('"run_vision_t4.py"')
+ assert cycles_at < spawn_at
+
+
+def test_a_vision_run_that_wrote_no_report_is_a_failure_not_a_silence():
+ """ "the vision run did not happen" and "the vision run passed" are opposite
+ outcomes, and an absent report must not read as the second."""
+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")
+ assert '"the vision process wrote no report"' in src
+ assert 'failures += report["vision_failures"]' in src
+
+
+def test_the_vision_step_count_is_pinned_low_rather_than_inherited():
+ """A vision step on a T4 is ~100s (317.9s for three, measured on
+ unsloth-probe-vision-train-r3). Inheriting the text side's --max-steps
+ would quietly add half an hour to the leg."""
+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")
+ spawn = src[src.index('"run_vision_t4.py"') :]
+ spawn = spawn[: spawn.index("subprocess.run(vision_cmd)")]
+ # Whitespace-insensitive: the repo's formatter reflows this list to one
+ # argument per line, and a guard matching the unformatted spelling goes red
+ # on a reformat rather than on a regression. That has now happened twice in
+ # this payload, so it is worth doing by default.
+ flat = "".join(spawn.split())
+ assert '"--max-steps","3",' in flat
+ assert "args.max_steps" not in flat, "the text step count must not reach the vision run"