Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.
Python agentaichatgptdeepseekfine-tuninggemmaimage-generationllamallmllmsopenaipythonqwenreinforcement-learningself-hostedstable-diffusiontext-to-speechttsuiunsloth
Kaggle T4: Latest_compile and Vision_FLA_compile legs, plus gpt-oss completions and GGUF (#9700)
Daniel Han40f452effaae
diff --git a/.github/scripts/kaggle_prefetch.py b/.github/scripts/kaggle_prefetch.pynew file mode 100644index 00000000000..958a4554162--- /dev/null+++ b/.github/scripts/kaggle_prefetch.py@@ -0,0 +1,260 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""The model prefetch that both Kaggle kernels paste into a generated cell.++A Kaggle GPU session is 2xT4 and the kernel keeps both busy training, but+downloading a model is CPU and network work that holds a card idle for its+whole duration. This module is the body of a lane that does that downloading+EARLY, beside the training legs, so the leg that needs the model finds it+already on disk.++Why a module and not a copy in each builder: `kaggle_t4_ci` prefetches the leg+models into the Kaggle image's default cache, and `kaggle_studio_ci` prefetches+Studio's two models into Studio's own private ``HF_HOME``. Same retry policy,+same reporting, two different cache roots. One copy that takes the root as an+argument is the only version of that which stays in agreement with itself.++Load it BY PATH (``importlib.util.spec_from_file_location``), never with a+plain ``import``. Both script directories already ship a ``build_kernel.py``+and a ``report.py``, the test suite puts both on ``sys.path``, and a plain+import therefore resolves to whichever reached ``sys.modules`` first -- which+is decided by test order rather than by intent. That collision has been paid+for here once already: one ``sys.path.insert`` added for a single test took+nine unrelated tests down with it.++WHAT THIS IS NOT: it is not a correctness mechanism. Every caller must treat a+failed prefetch as a no-op, because the payload that wants the model downloads+it for itself exactly as it did before this existed. A prefetch that fails the+kernel would be a new way to go red for something that is not under test.+"""++from __future__ import annotations++# The sentinel the driver and the reporters grep for. One record per repo, on+# its own line, so `kernel.log` ALONE measures the download -- which is the+# number the whole schedule is built around and the one thing no artifact has+# ever separated from weight-load time.+PREFETCH_SENTINEL = "KAGGLE_CI_PREFETCH"+++def _normalise(repos):+ """``["a", ("b", ["*.gguf"])]`` -> ``[("a", None), ("b", ["*.gguf"])]``.++ A bare string means the WHOLE repo, which is right for a small model whose+ every file gets loaded and wrong for anything with variants. Run+ 32667451396 fetched 69.1 GB of ``Qwen3.5-2B-GGUF`` -- every quant in the+ repo -- so that Studio could load one UD-Q4_K_XL file, and 55.1 GB of a+ checkpoint that was never opened at all. On a 4-core Kaggle box that is not+ just wasted bandwidth: it is CPU stolen from the payloads the prefetch+ exists to speed up, and it pushed the Studio install from 258s to 673.5s.+ """+ out = []+ for entry in repos:+ if isinstance(entry, str):+ out.append((entry, None))+ continue+ repo, patterns = entry+ out.append((repo, list(patterns) if patterns else None))+ return out+++def prefetch_cell(+ repos: list,+ *,+ hf_home: str | None = None,+ attempt_timeout: int = 900,+ total_timeout: int = 1800,+) -> str:+ """Source for a cell (or a driver thread) that warms ``repos``, in order.++ ``repos`` is ordered and the order is load bearing: the caller puts the+ repo with the longest lead time first, because a prefetch only pays for+ the work it finishes BEFORE the payload that wants it starts.++ ``hf_home`` of None means "do not touch HF_HOME", which is what the leg+ prefetch needs -- the legs read the Kaggle image's default cache and the+ entire point is to land in the cache they read. Setting it to a private+ directory there would produce a perfectly healthy prefetch that no payload+ can see, a full 12 GB of work thrown away, and a green run.+ """+ # repr(), NOT json.dumps(). This text is Python, and `json.dumps(None)` is+ # `null`, which parses fine and dies with a NameError the first time the+ # cell RUNS -- on a Kaggle session, minutes in, having already paid for the+ # box. `test_the_generated_prefetch_cell_runs` exists because compiling the+ # cell did not catch exactly that.+ return f'''+import json, os, threading, time++_REPOS = {_normalise(repos)!r}+_HF_HOME = {hf_home!r}+_ATTEMPT_TIMEOUT = {attempt_timeout}+_TOTAL_TIMEOUT = {total_timeout}+_DEADLINE = time.time() + _TOTAL_TIMEOUT++if _HF_HOME:+ os.environ["HF_HOME"] = _HF_HOME+++def _repo_bytes(repo):+ """Size of THIS repo's directory in the hub cache.++ Measured rather than taken from the return value, because+ `snapshot_download` reports a path and not a transfer size, and a repo that+ was ALREADY warm has to read as zero new bytes rather than as its own size+ or every rerun looks like a full download.++ Scoped to the one repo rather than differencing the whole cache, for two+ reasons. The cheap one: walking a cache holding a 12 GB model, twice per+ repo, is real time charged to the very measurement it is taking. The one+ that matters: the legs are downloading into this same cache CONCURRENTLY,+ so a whole-cache delta silently credits their bytes to this lane and+ reports a download rate the Hub never delivered.++ Counted ONCE per blob. A hub snapshot is a tree of symlinks into `blobs/`,+ so `os.stat` follows the link and the same bytes are then counted again+ under the blob's own name -- roughly 2x, which is not a rounding error in a+ number whose whole purpose is a MB/s rate. `lstat` measures the link+ itself, and the inode set also collapses hardlinks, which is what a+ filesystem without symlink support gives instead.+ """+ root = os.environ.get("HF_HOME") or os.path.expanduser("~/.cache/huggingface")+ folder = "models--" + repo.replace("/", "--")+ total = 0+ seen = set()+ for dirpath, _dirnames, filenames in os.walk(+ os.path.join(root, "hub", folder), followlinks = False+ ):+ for name in filenames:+ path = os.path.join(dirpath, name)+ if os.path.islink(path):+ continue+ try:+ info = os.lstat(path)+ except OSError:+ continue+ key = (info.st_dev, info.st_ino)+ if key in seen:+ continue+ seen.add(key)+ total += info.st_size+ return total+++def _attempt(repo, patterns, disable_xet):+ """One `snapshot_download`, in a thread, under a wall-clock watchdog.++ The watchdog is the point, and it is not the same thing as a retry. Xet+ classifies 408/429/5xx as transient and retries them itself with backoff+ (5 attempts, 3s base, a six-minute cap per delay), so a throttled or+ stalling transfer can sit inside ONE call for many minutes without ever+ raising anything for an ordinary `except` to catch. There are documented+ cases of Xet stalling where classic HTTP ran at line speed. So an attempt+ that stops making progress has to be abandoned on the clock and retried on+ a DIFFERENT transport; retrying the same stalling transport is how a retry+ loop turns into a way to spend the whole session.++ The thread is left running (daemon) rather than killed -- Python cannot+ kill it -- and its bytes are not lost: since huggingface_hub 1.18.0 each+ file lands through a process-unique temporary and an atomic move, and the+ files it already completed stay valid in the cache for the next attempt.+ """+ env_backup = os.environ.get("HF_HUB_DISABLE_XET")+ if disable_xet:+ os.environ["HF_HUB_DISABLE_XET"] = "1"+ box = {{}}++ def _run():+ try:+ from huggingface_hub import snapshot_download+ snapshot_download(repo_id=repo, allow_patterns=patterns)+ box["ok"] = True+ except BaseException as exc: # noqa: BLE001+ box["error"] = f"{{type(exc).__name__}}: {{exc}}"++ t = threading.Thread(target=_run, daemon=True)+ t.start()+ budget = min(_ATTEMPT_TIMEOUT, max(1.0, _DEADLINE - time.time()))+ t.join(budget)+ if env_backup is None:+ os.environ.pop("HF_HUB_DISABLE_XET", None)+ else:+ os.environ["HF_HUB_DISABLE_XET"] = env_backup+ if t.is_alive():+ return False, f"no result within {{budget:.0f}}s"+ if box.get("ok"):+ return True, None+ return False, box.get("error", "unknown")+++def prefetch_all():+ for repo, patterns in _REPOS:+ started = time.time()+ before = _repo_bytes(repo)+ ok, error, transport, attempts = False, None, None, 0+ # Time inside the attempt that WORKED, separately from the wall clock+ # of the whole repo. They differ by the backoff sleeps and the failed+ # attempts, and conflating them corrupts the one number this lane+ # exists to produce: a run that retried twice would report its 12 GB as+ # having taken 9s longer than it did and understate the achieved rate+ # accordingly. `seconds` is what the SCHEDULE waits for; `download_+ # seconds` is how fast the Hub actually was.+ download_seconds = None+ # Last attempt forces classic HTTP. The first two keep whatever+ # transport the hub chose (Xet when the repo has Xet metadata and+ # hf_xet is installed, which since huggingface_hub 0.32 is the default+ # and is therefore live here whether or not anyone chose it).+ _plan = (False, False, True)+ for _i, disable_xet in enumerate(_plan):+ if time.time() >= _DEADLINE:+ error = "the prefetch budget was spent before this repo"+ break+ attempts += 1+ _t0 = time.time()+ ok, error = _attempt(repo, patterns, disable_xet)+ transport = "http" if disable_xet else "auto"+ if ok:+ download_seconds = round(time.time() - _t0, 1)+ break+ print(f"{PREFETCH_SENTINEL}_RETRY " + json.dumps(+ {{"repo": repo, "attempt": attempts, "error": str(error)[:300]}}),+ flush=True)+ # Backoff only when something will actually follow it. Sleeping+ # after the LAST attempt buys nothing -- there is no retry left to+ # space out -- and it is the difference between this lane giving up+ # in 9s and giving up in 18s, per repo, on the path where the+ # network is already known to be unhappy.+ #+ # Bounded by what is LEFT of the budget too, not just by the curve:+ # a lane already at its deadline that still sleeps 15s is spending+ # session time to accomplish nothing.+ if _i + 1 < len(_plan):+ time.sleep(max(0.0, min(15.0, 3.0 * attempts, _DEADLINE - time.time())))+ seconds = round(time.time() - started, 1)+ moved = max(0, _repo_bytes(repo) - before)+ print(f"{PREFETCH_SENTINEL} " + json.dumps({{+ "repo": repo, "ok": bool(ok), "seconds": seconds,+ "download_seconds": download_seconds, "bytes": moved,+ "mb_per_s": (round(moved / 1e6 / download_seconds, 1)+ if download_seconds else None),+ # Reported so an over-narrow filter is visible rather than silent.+ # A pattern that matches nothing downloads nothing, reports ok and+ # leaves the payload to fetch the model itself -- a prefetch that+ # looks perfect and does nothing. `bytes` next to `patterns` is+ # what makes that readable in the summary.+ "patterns": patterns,+ "transport": transport, "attempts": attempts,+ "error": None if ok else str(error)[:300],+ }}), flush=True)+++# Never raises. A prefetch is an optimisation: the payload that wants the model+# still downloads it for itself, so a failure here costs seconds, and letting+# it propagate would invent a way for the kernel to go red for something that+# is not under test.+try:+ prefetch_all()+except BaseException as exc: # noqa: BLE001+ print(f"{PREFETCH_SENTINEL}_ABORTED " + json.dumps(+ {{"error": f"{{type(exc).__name__}}: {{exc}}"}}), flush=True)+'''diff --git a/.github/scripts/kaggle_studio_ci/build_kernel.py b/.github/scripts/kaggle_studio_ci/build_kernel.pyindex f5b187d3322..6ef1a51ed15 100644--- a/.github/scripts/kaggle_studio_ci/build_kernel.py+++ b/.github/scripts/kaggle_studio_ci/build_kernel.py@@ -115,8 +115,106 @@ def _code_cell(source: str) -> dict:}-def build_payload_notebook(*, unsloth_ref: str, repo_url: str, payload_args: str) -> dict:- """The notebook that installs Unsloth and runs the payload against it."""+def _prefetch_builder():+ """Load ``kaggle_prefetch.py`` by PATH. See the note in+ ``kaggle_t4_ci/build_kernel.py``: two sibling script directories both ship+ a ``build_kernel`` and a ``report``, so a plain import here resolves by+ whichever landed in ``sys.modules`` first.+ """+ import importlib.util++ path = Path(__file__).resolve().parents[1] / "kaggle_prefetch.py"+ spec = importlib.util.spec_from_file_location("kaggle_ci__prefetch", path)+ if spec is None or spec.loader is None:+ raise ImportError(f"cannot load the prefetch builder from {path}")+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return module+++def _models_from(payload_args: str) -> list[str]:+ """The repos this payload will load, read off its own argv.++ NOT a second copy of the defaults. ``--chat-model`` and ``--train-model``+ are dispatch inputs, so a hardcoded pair here would prefetch the wrong+ models the moment anyone used them -- and prefetching the wrong repo is+ invisible: it downloads happily, warms a cache nobody reads, and reports+ success.+ """+ tokens = payload_args.split()+ picked = {}+ for flag, default in (+ # These defaults must track run_studio_gpu.py's own, or the prefetch+ # warms a cache the payload never reads -- which downloads happily and+ # reports success. tests/kaggle/test_t4_ci_transport.py compares the+ # two, which is how this pair was caught drifting.+ ("--chat-model", "unsloth/Qwen3.5-2B-MTP-GGUF"),+ ("--train-model", "unsloth/Qwen3.5-2B"),+ # Read for the same reason as the repos: Studio loads ONE quant out of+ # a GGUF repo that ships many, and an unfiltered snapshot pulls all of+ # them. Run 32667451396 fetched 69.1 GB of Qwen3.5-2B-GGUF to serve a+ # single UD-Q4_K_XL file. Taken off argv rather than hardcoded so a+ # dispatch that overrides the variant filters on the variant it chose.+ ("--chat-variant", "UD-Q4_K_XL"),+ ):+ value = default+ for i, token in enumerate(tokens):+ if token == flag and i + 1 < len(tokens):+ value = tokens[i + 1]+ elif token.startswith(flag + "="):+ value = token.split("=", 1)[1]+ picked[flag] = value+ # Chat model first: it is the GGUF that llama.cpp has to serve, and it is+ # the larger of the two.+ #+ # The variant glob is deliberately loose at both ends. Multi-part GGUFs are+ # named `...UD-Q4_K_XL-00001-of-00002.gguf`, so anchoring the suffix would+ # match the single-file case and silently miss every shard of the split+ # one -- which downloads nothing, reports success, and leaves Studio to+ # fetch it itself.+ variant = picked["--chat-variant"]+ chat = (picked["--chat-model"], [f"*{variant}*"]) if variant else picked["--chat-model"]+ return [chat, picked["--train-model"]]+++def build_payload_notebook(+ *,+ unsloth_ref: str,+ repo_url: str,+ payload_args: str,+ phase: str | None = None,+) -> dict:+ """The notebook that installs Unsloth and runs the payload against it.++ ``phase`` splits that notebook in two, for the merged kernel that runs this+ payload beside the T4 notebook legs (see ``kaggle_t4_ci/build_kernel.py``).+ The split point is not arbitrary: everything up to and including the+ Playwright install is checkout, download and compile, none of which touches+ a GPU, and the ``verify`` cell is the first thing that requires one -- it+ refuses with "no CUDA device in the Studio venv" when+ ``torch.cuda.is_available()`` is False. So++ * ``"install"`` is the GPU-free prefix and can run while both cards are+ busy training,+ * ``"test"`` is everything that needs a card, and runs once they are free.++ ``None`` builds the whole thing as one notebook, which is what the+ standalone Studio workflow still does.++ The two halves communicate through the DISK, not through the interpreter:+ ``setup`` recomputes the same paths in both (``_pick_work_root`` is+ deterministic within a session) and the test half re-derives ``VENV_PY``+ from ``STUDIO_HOME`` rather than inheriting it.++ One trap that is easy to walk into here: the install half must still SEE+ both GPUs. ``install.sh --local`` resolves torch, and a CPU-only torch+ resolved by an installer that could not find a device is precisely the+ regression the verify cell exists to catch. So the caller leaves+ ``CUDA_VISIBLE_DEVICES`` unset on that lane rather than blanking it; the+ install reads device capability and never allocates.+ """+ if phase not in (None, "install", "test"):+ raise ValueError(f"phase must be None, 'install' or 'test', not {phase!r}")setup = f"""# Where everything lives.#@@ -338,15 +436,60 @@ print("{PAYLOAD_SENTINEL} complete rc=" + str(proc.returncode), flush=True)# aborting here would lose the cells below it."""+ # Studio's two models, fetched on the half that is ALREADY hidden.+ #+ # Both were previously pulled inside run_studio_gpu.py, which is the TEST+ # half, so the merged kernel hid Studio's clone, pip and Playwright browser+ # and then paid full price for its downloads with both cards idle. They go+ # here instead, under Studio's own HF_HOME -- which is why this cannot use+ # the t4 driver's lane: that one deliberately targets the image default so+ # the training legs can read it, and Studio's install is a user-shaped+ # install with a cache root of its own.+ #+ # Last in the install phase, after the venv and the browser: those are what+ # the test half cannot start without, and a download that overruns the card+ # queue must not be what delays them.+ #+ # hf_home=None means "inherit", NOT "use the default". The setup cell runs+ # first in this same notebook and has already put Studio's private root in+ # os.environ["HF_HOME"], so inheriting is how this lands there. Passing the+ # path again would be a second copy of _pick_work_root's answer, free to+ # disagree with the real one. `test_the_studio_prefetch_lands_in_studios+ # _own_cache` pins the ordering that makes inheriting correct.+ prefetch = _prefetch_builder().prefetch_cell(+ _models_from(payload_args),+ hf_home = None,+ attempt_timeout = 600,+ total_timeout = 1200,+ )++ # Marks the GPU-free half done, on its own line, so the driver can gate the+ # test half on a sentinel it saw rather than on a returncode alone.+ installed = f"""print("{PAYLOAD_SENTINEL} INSTALLED " + json.dumps({{+ "studio_home": str(STUDIO_HOME), "venv": str(VENV_PY),+}}), flush=True)+"""++ # Re-derives what the install half left on disk. VENV_PY is defined in the+ # install cell, which the test half does not carry, so without this the+ # verify cell below dies on a NameError rather than on anything it tests.+ bridge = f"""VENV_PY = STUDIO_HOME / "unsloth_studio" / "bin" / "python"+if not VENV_PY.is_file():+ fail_report(f"the install phase left no interpreter at {{VENV_PY}}; it either "+ f"did not run or did not land in the directory this half looks in")+ raise SystemExit(f"no interpreter at {{VENV_PY}}")+print("{PAYLOAD_SENTINEL} venv " + str(VENV_PY), flush=True)+"""++ if phase == "install":+ cells = [setup, clone, install, browser, prefetch, installed]+ elif phase == "test":+ cells = [setup, bridge, verify, run]+ else:+ cells = [setup, clone, install, browser, prefetch, verify, run]+return {- "cells": [- _code_cell(setup),- _code_cell(clone),- _code_cell(install),- _code_cell(browser),- _code_cell(verify),- _code_cell(run),- ],+ "cells": [_code_cell(source) for source in cells],"metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},"language_info": {"name": "python"},diff --git a/.github/scripts/kaggle_studio_ci/report.py b/.github/scripts/kaggle_studio_ci/report.pyindex b64eebdfaf3..916fb97b7c9 100644--- a/.github/scripts/kaggle_studio_ci/report.py+++ b/.github/scripts/kaggle_studio_ci/report.py@@ -141,9 +141,46 @@ def render(report: dict) -> list[str]:return lines+# The label this payload reports under. Duplicated in kaggle_t4_ci/report.py;+# see the note there for why it is not shared.+STUDIO_LABEL = "studio-gpu"+++def own_verdict(kernel_verdict: str, kernel_reason: str, reports: list, expect: int):+ """This reporter's verdict over ITS OWN payloads, not the kernel's.++ The launcher writes one verdict for the whole kernel, and since+ --with-studio that kernel holds two unrelated experiments. Reading the+ kernel verdict here means a failing training leg prints "Studio GPU smoke: FAIL"+ above a section listing zero failures, and a failing Studio payload prints+ the same over four green legs. Both are the misleading-red twin of the+ green tick that tested nothing, and both would send someone to read the+ wrong payload.++ So the verdict is recomputed from the filtered reports. The kernel reason+ is kept only when the two agree; otherwise it describes the other half.++ `infra` is deliberately not synthesised: with nothing of ours back, the+ kernel-level reason (quota, concurrency cap, a push that was throttled) is+ the only account of why, and it applies to every payload equally.+ """+ if not reports:+ return (kernel_verdict if kernel_verdict == "infra" else "partial"), kernel_reason+ failing = [r for r in reports if not r.get("passed")]+ if failing:+ return "fail", f"{len(failing)} of {len(reports)} payload(s) failed their assertions"+ if len(reports) < expect:+ return "partial", f"only {len(reports)} of {expect} payload(s) reported back"+ return "pass", f"all {len(reports)} payload(s) passed"++def main() -> int:ap = argparse.ArgumentParser()ap.add_argument("--evidence", required = True)+ # One, always: the payload's two notebooks are halves of one experiment and+ # produce exactly one `studio-gpu` report between them. See the count note+ # in kaggle_t4_ci/build_kernel.py's --all-kernels tail.+ ap.add_argument("--expect", type = int, default = 1)args = ap.parse_args()evidence = Path(args.evidence)@@ -162,6 +199,15 @@ def main() -> int:reason = result.get("reason", "")reports = result.get("reports", [])+ # This payload can share a kernel with the T4 notebook legs (see+ # kaggle_t4_ci/build_kernel.py --with-studio), and every payload in that+ # kernel reports through the same prefix. The legs are a different shape --+ # a per-step metric trace rather than assertions -- so rendering them here+ # would produce Studio sections describing training runs. Each reporter+ # owns its own labels.+ reports = [r for r in reports if r.get("label") == STUDIO_LABEL]+ verdict, reason = own_verdict(verdict, reason, reports, args.expect)+header = {"pass": "### Unsloth GPU smoke: PASS","fail": "### Unsloth GPU smoke: FAIL",diff --git a/.github/scripts/kaggle_t4_ci/build_kernel.py b/.github/scripts/kaggle_t4_ci/build_kernel.pyindex 9bc646266e9..c17ee5daff3 100644--- a/.github/scripts/kaggle_t4_ci/build_kernel.py+++ b/.github/scripts/kaggle_t4_ci/build_kernel.py@@ -56,7 +56,14 @@ from pathlib import Pathsys.path.insert(0, str(Path(__file__).resolve().parent))-from legs import KERNELS, PACKAGE_UNDER_TEST, Leg, expand_install, resolve # noqa: E402+from legs import ( # noqa: E402+ KERNELS,+ PACKAGE_UNDER_TEST,+ PREFETCH_REPOS,+ Leg,+ expand_install,+ resolve,+)DRIVER_SENTINEL = "KAGGLE_T4_CI_DRIVER"PAYLOAD_SENTINEL = "KAGGLE_T4_CI_PAYLOAD"@@ -145,8 +152,16 @@ def build_payload_notebook(extra_args: tuple[str, ...] = (),reference: str | None = None,) -> dict:- """A single-GPU notebook that installs one leg's stack and runs it."""+ """A notebook that installs one leg's stack and runs it.++ One GPU for every leg but the `all_cards` one, which is unpinned so that+ unsloth's DEVICE_COUNT > 1 code path is reachable at all.+ """root = _kernel_root(leg)+ # Baked in HERE, at build time, from the leg's own declaration. The check+ # in the verify cell compares against this constant rather than against+ # anything the running kernel can see.+ expected_visible = 2 if leg.all_cards else 1wanted = list(leg.files)if leg.entry not in wanted:wanted.append(leg.entry)@@ -185,12 +200,45 @@ print("{PAYLOAD_SENTINEL} sources " + json.dumps(sorted(FILES)), flush=True)# differs between the control leg and the version canary. They are printed# before they run so the kernel log alone says what was asked for, which is# what makes a canary failure attributable without downloading anything.-import json, subprocess, sys, time+import glob, json, os, re, subprocess, sys, timeprint("{PAYLOAD_SENTINEL} leg {leg.name}: {leg.summary}", flush=True)GROUPS = {json.dumps(groups)}+UNINSTALL = {json.dumps(list(leg.uninstall))}print("{PAYLOAD_SENTINEL} install plan " + json.dumps(GROUPS), flush=True)+# Wheels the driver built ONCE, before any leg started, from the very SHAs+# these groups name. Empty or missing means the build did not happen or did not+# finish, and every spec below then resolves exactly as it did before -- a+# failed wheel build costs its own time and changes nothing about what is+# tested.+WHEEL_DIR = os.environ.get("UNSLOTH_CI_WHEEL_DIR", "")+++# `name @ git+...@sha` -> the prebuilt wheel for `name`, when there is one.+#+# Matched on the DISTRIBUTION NAME, normalised the way a wheel filename is+# (PEP 503: runs of -_. collapse to _), because `unsloth_zoo` on the left of the+# `@` arrives as `unsloth_zoo-2026.8.1-py3-none-any.whl` on disk. Getting that+# wrong is invisible: no wheel matches, every leg falls back to its git spec,+# and the run is merely as slow as it was before.+#+# A comment and not a docstring: this whole cell is generated from an f-string+# delimited by triple quotes, so a docstring here ENDS the template and the+# build dies with a SyntaxError in build_kernel.py rather than in the cell.+def _localise(spec):+ if not WHEEL_DIR or "@ git+" not in spec:+ return spec+ name = spec.split("@ git+", 1)[0].strip()+ key = re.sub(r"[-_.]+", "_", name).lower()+ for cand in sorted(glob.glob(os.path.join(WHEEL_DIR, "*.whl"))):+ stem = os.path.basename(cand).split("-", 1)[0]+ if re.sub(r"[-_.]+", "_", stem).lower() == key:+ return cand+ return spec++def pip(args):+ args = [_localise(a) for a in args]cmd = [sys.executable, "-m", "pip", "install", "-q", *args]print(" $ " + " ".join(cmd[3:]), flush=True)# github.com occasionally 500s on a git fetch, and PyPI occasionally@@ -228,7 +276,35 @@ def pip(args):raise SystemExit(f"pip install failed: {{args}}")for group in GROUPS:+ _g0 = time.time()pip(group)+ # Per group, because "install took 191s" was all any artifact ever said and+ # it is not actionable: it hides which of the four groups was the cost, and+ # therefore whether the shared wheels helped at all.+ print("{PAYLOAD_SENTINEL} install group " + json.dumps({{+ "seconds": round(time.time() - _g0, 1),+ "spec": [_localise(a) for a in group],+ }}), flush=True)+# After the install groups, never before: these distributions arrive as+# dependencies of something installed above, so removing them earlier would+# remove nothing and the later install would put them back.+#+# Never fatal. If a name is absent pip says so and exits 0, and even a genuine+# failure here should not cost the session -- the payload then fails on its own+# terms with an error that names what actually went wrong, which is more useful+# than a stand-down during setup.+if UNINSTALL:+ _u0 = time.time()+ _up = subprocess.run(+ [sys.executable, "-m", "pip", "uninstall", "-y", *UNINSTALL],+ capture_output=True, text=True)+ print("{PAYLOAD_SENTINEL} uninstall " + json.dumps({{+ "seconds": round(time.time() - _u0, 1),+ "spec": UNINSTALL,+ "returncode": _up.returncode,+ "stdout": _up.stdout[-2000:],+ "stderr": _up.stderr[-2000:],+ }}), flush=True)print("{PAYLOAD_SENTINEL} install done", flush=True)"""@@ -325,12 +401,23 @@ try:"name": torch.cuda.get_device_name(0) if torch.cuda.is_available() else None,"visible": __import__("os").environ.get("CUDA_VISIBLE_DEVICES"),}}), flush=True)- # Exactly one GPU must be visible: two would mean the driver failed to pin- # this payload to its own card, and accelerate would shard across both,- # which is a different test from the single T4 a Colab user gets. Zero- # means the payload cannot run at all on the hardware it was billed for.- assert torch.cuda.device_count() == 1, (- f"expected exactly 1 visible GPU, got {{torch.cuda.device_count()}}")+ # EXACTLY the count this leg asked for, and the number is baked in at build+ # time rather than read from the environment -- reading it from the same+ # variable that sets it would make the check agree with whatever happened.+ #+ # 1 for every ordinary leg: two would mean the driver failed to pin this+ # payload to its own card, and accelerate would shard across both, which is+ # a different test from the single T4 a Colab user gets. Zero means the+ # payload cannot run at all on the hardware it was billed for.+ #+ # 2 for the `multi_gpu` leg, which is deliberately unpinned so that+ # unsloth's DEVICE_COUNT > 1 branch is reachable. A leg that asks for two+ # and gets one still fails here, which is the case that matters: silently+ # running it on one card would leave the multi-GPU claim asserted and+ # untested, and green.+ assert torch.cuda.device_count() == {expected_visible}, (+ f"expected exactly {expected_visible} visible GPU(s), got "+ f"{{torch.cuda.device_count()}}")except BaseException as exc:detail = f"{{type(exc).__name__}}: {{exc}}"print("{PAYLOAD_SENTINEL} GPU_UNUSABLE " + json.dumps(detail), flush=True)@@ -494,6 +581,14 @@ def build_driver(per_run_timeout: int,isolation: dict[str, bool] | None = None,expected_gpus: int = SESSION_GPUS,+ cpu_lane: str | None = None,+ after_gpu: str | None = None,+ prefetch_repos: tuple[str, ...] = (),+ vram_source: dict | None = None,+ after_gpu_concurrent: bool = False,+ shared_wheel_specs: tuple[str, ...] = (),+ overlays: dict[str, tuple[str, ...]] | None = None,+ all_card: tuple[str, ...] = (),) -> dict:"""Kernel notebook that runs the payloads across the session's GPUs.@@ -504,23 +599,84 @@ def build_driver(``payloads`` may hold MORE entries than there are cards. They queue: oneworker per card takes the next leg only when its current one has exited, soa card carries exactly one payload at a time. ``payloads`` is ordered, and- that order is the start order -- longest leg first, since the longest leg- is what sets the makespan and a greedy scheduler cannot balance around one- it picks up last.+ that order is the start order. It comes from ``legs.KERNELS``, which+ documents why gptoss sits third rather than first; do not sort it here.++ ``prefetch_repos`` are warmed, in the order given, on a lane that takes no+ card and no virtualenv. It runs in the driver's own interpreter using the+ Kaggle image's ``huggingface_hub``, so it starts at t=0 rather than after a+ venv build, and it deliberately does NOT set ``HF_HOME``: the legs read the+ image default and the whole point is to land in the cache they read.``expected_gpus`` is how many cards the packing was built for. It is whatthe kernel stands down against, and it is deliberately NOT ``len(payloads)``any more: see the guard in the generated cell."""+ vram_source = vram_source or {}encoded = {name: _encode_bytes(json.dumps(nb).encode("utf-8")) for name, nb in payloads.items()}isolation = isolation or {}system_site = {name: bool(isolation.get(name, True)) for name in payloads}- order = list(payloads)+ # Only legs carry overlays; Studio's halves install into their own tree.+ overlay_specs = {+ name: tuple(specs) for name, specs in (overlays or {}).items() if specs and name in payloads+ }+ # The CARD queue, which is not every payload. cpu_lane runs beside the+ # cards and after_gpu runs once they are free; leaving either in here would+ # hand it a card and defeat the point of both.+ off_queue = {name for name in (cpu_lane, after_gpu) if name}+ for name in off_queue:+ if name not in payloads:+ raise KeyError(f"{name!r} is scheduled off the card queue but is not a payload")+ order = [name for name in payloads if name not in off_queue]+ if not order:+ raise ValueError("every payload is off the card queue, so nothing would use a GPU")+ # hf_home is left unset ON PURPOSE. See build_driver's docstring: a private+ # root here downloads 12 GB into a cache no leg reads and reports success.+ # Per-payload VRAM for the admission check. Off-queue payloads are absent+ # on purpose: neither takes a card, so neither has a budget to consume.+ vram_gb = {name: float(getattr(leg, "vram_gb", 1.0)) for name, leg in vram_source.items()}+ # Studio's halves are not legs and have no Leg record. The test half is the+ # only one that takes a card, and it is priced from what it loads: a 2B+ # chat model at UD-Q4_K_XL plus a 0.5B LoRA. Deliberately generous -- an+ # under-price here is what would let it share with something that does not+ # fit, and the whole admission check is only as honest as its inputs.+ for name in (cpu_lane, after_gpu):+ if name:+ vram_gb.setdefault(name, 2.2)+ prefetch_blob = ""+ if prefetch_repos:+ prefetch_blob = _encode_bytes(+ _prefetch_builder().prefetch_cell(list(prefetch_repos)).encode("utf-8")+ )setup = f"""import base64, gzip, json, os, pathlib, subprocess, sys, threading, timeprint("{DRIVER_SENTINEL} start", flush=True)WORK = pathlib.Path("/kaggle/working")++# Virtualenvs do NOT live under WORK. /kaggle/working is 19.5 GB and is also+# what Kaggle ships home as the artifact; $HOME and /tmp share a ~1 TB overlay.+# Each leg's venv carries its own torch and its own NVIDIA runtime, and with+# two legs per card there can now be FOUR of them alive at once -- which does+# not fit in 19.5 GB and would surface as an install failing halfway through+# for reasons that look nothing like a disk.+#+# Evidence stays on WORK, because that is the only thing that has to come back.+VENV_ROOT = pathlib.Path("/tmp/t4ci_venvs")+try:+ VENV_ROOT.mkdir(parents=True, exist_ok=True)+except OSError:+ # A box without a writable /tmp is not one this kernel can fix. Falling+ # back to WORK keeps a one-leg-per-card run working, which is the shape+ # that fitted there before co-scheduling.+ VENV_ROOT = WORK+print("{DRIVER_SENTINEL}_VENV_ROOT " + json.dumps({{"path": str(VENV_ROOT)}}), flush=True)+# Defined HERE, not beside the build below. run_one exports it into every+# payload's env, and the Studio CPU lane calls run_one before the build block+# is reached -- so assigning it there is a NameError on the first payload to+# start, which is the one that runs earliest and would take the whole kernel+# down with it.+WHEEL_DIR = VENV_ROOT / "wheels"PAYLOADS = {json.dumps(encoded)}# Which payloads may see the Kaggle image's site-packages. A leg that# replaces torch must not: pip would then treat torch's pinned NVIDIA@@ -532,6 +688,18 @@ PAYLOADS = {json.dumps(encoded)}# _defines is what caught that, which is the whole reason it exists.SYSTEM_SITE = {system_site!r}+# Per-leg pure-Python version pins, laid OVER the venv rather than resolved+# into it. See Leg.overlay. Empty for a leg that wants the base as it resolved.+OVERLAYS = {overlay_specs!r}+# Distributions an overlay may never carry. torch, triton and bitsandbytes are+# native and coupled to the driver stack: a shadowed copy on PYTHONPATH either+# fails to import or, worse, imports and silently disagrees with the CUDA+# runtime already loaded. The overlay exists to move transformers/trl/peft, not+# to rebuild the machine. Substring match, lowercased, so `nvidia-cublas-cu12`+# and `torchvision` are caught without listing every wheel NVIDIA ships.+OVERLAY_DENY = ("torch", "triton", "bitsandbytes", "nvidia", "cuda",+ "xformers", "flash")+# How many GPUs did we actually get? Assert rather than assume: a 1-GPU# allocation would silently serialise everything onto device 0 and double# every runtime while looking like a slow but healthy run.@@ -551,6 +719,72 @@ N_GPU = len(GPUS)# legs.KERNELS, so the packing decision lives beside the legs it packs.ORDER = {order!r}+# Requirement specs that EVERY leg installs identically, built once up front.+# Derived from the legs' own install groups rather than written out again here,+# because a second copy of the two SHAs is a second thing to keep in step -- and+# one that drifts silently, since a wheel built from the wrong ref installs+# perfectly.+SHARED_WHEEL_SPECS = {shared_wheel_specs!r}++# Payloads that are NOT in the card queue.+#+# CPU_LANE runs from t=0 alongside the training legs and never takes a card.+# AFTER_GPU runs once the queue has drained and wants every card. Both are None+# on an ordinary notebook-only kernel, which leaves the schedule exactly as it+# was.+CPU_LANE = {cpu_lane!r}+AFTER_GPU = {after_gpu!r}+# ALL_CARD payloads want EVERY card visible rather than one. They are not in+# the card queue either, because the queue hands out one card at a time and+# there is no index to give them.+#+# They are NOT the same as AFTER_GPU. AFTER_GPU is unpinned AND waits for the+# queue to drain; an ALL_CARD leg reserves its (small) share on every card+# through _admit_all and then runs BESIDE the training legs, which is what+# makes covering unsloth's DEVICE_COUNT > 1 branch cost nothing.+ALL_CARD = {all_card!r}+# When true, AFTER_GPU does not wait for the cards to drain. It is admitted to+# a card by the same VRAM check the legs use, so it runs BESIDE a light leg+# rather than behind the last one.+#+# Worth ~211s in simulation, because Studio's install-then-test chain is the+# critical path while the legs have slack. What it costs is real and is why it+# is a flag and not the default: Studio then sees ONE card instead of two, and+# its own device selection is part of what it exists to prove.+AFTER_GPU_CONCURRENT = {after_gpu_concurrent!r}++# Measured peak_reserved_gb per payload (legs.Leg.vram_gb), and the budget a+# single card will admit. A T4 reports 14.56 GB usable; the budget is 13.0 so+# that fragmentation and the allocator's own headroom are not the thing that+# discovers the limit.+#+# This is what lets two legs share a card. On run 32611343797 the three Qwen+# legs peaked at 0.70 GB EACH -- 4.8% of the card -- while gptoss peaked at+# 12.78 GB, so "one payload per card" was leaving a card 95% empty for three+# quarters of the run. gptoss still never shares: 12.78 + 0.70 = 13.48 is over+# budget, which is the arithmetic rather than a special case.+#+# MAX_LEGS_PER_CARD caps it at 2 even where VRAM would allow 3, because VRAM is+# not the scarce resource here. A Kaggle session has 4 vCPUs and a leg is ~87%+# install, import and download; the legs contend for CORES long before they+# contend for memory, and a third concurrent install buys nothing the profile+# says is available.+VRAM_GB = {vram_gb!r}+CARD_VRAM_BUDGET_GB = 13.0+# 1 when the venvs had to fall back to WORK. The fallback above says it "keeps+# a one-leg-per-card run working", and until this line that was a description+# of an intention rather than of anything the code did: MAX_LEGS_PER_CARD was a+# constant, so a box with no writable /tmp would still build four torch venvs,+# two at a time per card, in the 19.5 GB /kaggle/working that the comment at+# the top of this block says they do not fit in. It would have surfaced as an+# install failing halfway through for reasons that look nothing like a disk.+MAX_LEGS_PER_CARD = 2 if VENV_ROOT != WORK else 1++# The model prefetch, gzip+base64 like the payloads so its quoting survives+# being embedded in a generated cell. Empty string means no prefetch, which is+# what every kernel did before this existed.+PREFETCH_BLOB = {prefetch_blob!r}+# A shortfall is INFRASTRUCTURE, and it has to be called that HERE, before a# thread starts. `max(1, ...)` used to make one card look like enough: both# payloads were pinned to device 0, each child still saw exactly one GPU and@@ -584,7 +818,7 @@ lock = threading.Lock()def _make_venv(idx, system_site):"""Give a child its own site-packages. See this file's module docstring."""- vdir = WORK / f"venv_{{idx}}"+ vdir = VENV_ROOT / f"venv_{{idx}}"try:uv = subprocess.run(["which", "uv"], capture_output=True,text=True).stdout.strip()@@ -592,7 +826,23 @@ def _make_venv(idx, system_site):subprocess.run([sys.executable, "-m", "pip", "install", "-q", "uv"],check=True, timeout=900)uv = "uv"- venv_cmd = [uv, "venv", str(vdir), "--seed"]+ # PIN THE INTERPRETER TO THE DRIVER'S OWN, and it is not cosmetic.+ #+ # uv's default python-preference is `managed`, so once ANY managed+ # CPython exists under ~/.local/share/uv/python -- which something in a+ # longer kernel eventually downloads -- a bare `uv venv` silently builds+ # on THAT interpreter instead of the image's. `--system-site-packages`+ # is still honoured and still inherits nothing, because a 3.13 venv+ # cannot see a 3.12 site-packages.+ #+ # Measured, 6 sessions, deterministic per arm: the Default leg came up+ # on python 3.13.13 / torch 2.12.1+cu130 / datasets 5.0.1 in all three+ # arm-B runs and on 3.12.13 / torch 2.10.0+cu128 / datasets 4.3.0 in+ # all three arm-A runs, from one commit. Every other leg stayed on+ # 3.12. The 3.13 runs then failed 3/3 in `Dataset.from_dict` -- see the+ # overlay comment below for why -- and the whole leg was resolving a+ # stack no Kaggle user has.+ venv_cmd = [uv, "venv", str(vdir), "--seed", "--python", sys.executable]if system_site:venv_cmd.append("--system-site-packages")subprocess.run(venv_cmd, check=True, timeout=900)@@ -602,8 +852,16 @@ def _make_venv(idx, system_site):kname = f"t4ci{{idx}}"subprocess.run([str(py), "-m", "ipykernel", "install", "--user","--name", kname], check=True, timeout=600)+ # The version is REPORTED, not assumed. A venv on the wrong interpreter+ # is invisible in the driver log otherwise, and it was: three sessions+ # went red before anyone read a python version out of a leg's report.+ _ver = subprocess.run(+ [str(py), "-c", "import sys;print('%d.%d.%d' % sys.version_info[:3])"],+ capture_output=True, text=True, timeout=120).stdout.strip()print(f"{DRIVER_SENTINEL}_VENV " + json.dumps(- {{"idx": idx, "kernel": kname, "system_site": system_site}}),+ {{"idx": idx, "kernel": kname, "system_site": system_site,+ "python": _ver,+ "driver_python": "%d.%d.%d" % sys.version_info[:3]}}),flush=True)return str(py), kname, str(vdir / "bin")except Exception as exc:@@ -617,7 +875,17 @@ def run_one(name, gpu_index, idx):env = dict(os.environ)# One GPU per payload. This is what makes the run comparable to the# single T4 a Colab user gets; two visible GPUs is a different test.- env["CUDA_VISIBLE_DEVICES"] = str(gpu_index)+ #+ # gpu_index None means DO NOT PIN, and it is not the same as pinning to+ # nothing. It is used by the Studio lanes: the install half must still see+ # both cards, because install.sh --local resolves torch and a CPU-only+ # torch resolved by an installer that could not find a device is the exact+ # regression Studio's verify cell exists to catch; and the test half wants+ # both because Studio's own device selection is part of what it tests, so+ # masking one would test a machine nobody has. Blanking the variable would+ # do the opposite of both.+ if gpu_index is not None:+ env["CUDA_VISIBLE_DEVICES"] = str(gpu_index)env["PYTHONUNBUFFERED"] = "1"# One compile cache per payload. `unsloth_compiled_cache` is a RELATIVE# path resolved against the working directory, which both papermill@@ -629,6 +897,41 @@ def run_one(name, gpu_index, idx):env["UNSLOTH_COMPILE_LOCATION"] = str(WORK / ("unsloth_compiled_cache_" + pathlib.Path(name).stem))+ # The OTHER three caches, which a separate venv does not protect.+ #+ # UNSLOTH_COMPILE_LOCATION above covered unsloth's generated modules and+ # was assumed to be the whole story. It is not. torch and triton key their+ # caches off $TMPDIR, not off the interpreter, so on torch 2.9.1 an unset+ # TORCHINDUCTOR_CACHE_DIR resolves to `tempfile.gettempdir()/torchinductor_+ # $USER` (torch/_inductor/runtime/cache_dir_utils.py:22) and the triton+ # cache lands UNDER that same directory. Every leg on the box therefore+ # shared /tmp/torchinductor_root -- four payloads whose entire purpose is+ # to install DIFFERENT transformers/TRL/peft versions and compile the same+ # modules, writing one cache between them.+ #+ # Per-leg directories rather than TORCHINDUCTOR_FORCE_DISABLE_CACHES or+ # TRITON_ALWAYS_COMPILE, which are the blunter knobs and do exist+ # (torch/compiler/config.py:85 and triton/knobs.py:358). Those force every+ # leg to recompile from cold, which spends time on the critical path this+ # kernel is being shortened along, and makes the legs measure a machine no+ # user has. Isolation is what is wanted here, not cache defeat.+ #+ # TMPDIR is set too, and it is the belt to the braces: anything else that+ # resolves a cache through gettempdir() follows it without this file having+ # to know the variable's name.+ _leg_tmp = VENV_ROOT / ("tmp_" + pathlib.Path(name).stem)+ for _sub in ("", "inductor", "triton"):+ try:+ (_leg_tmp / _sub).mkdir(parents=True, exist_ok=True)+ except OSError:+ pass+ env["TMPDIR"] = str(_leg_tmp)+ env["TORCHINDUCTOR_CACHE_DIR"] = str(_leg_tmp / "inductor")+ env["TRITON_CACHE_DIR"] = str(_leg_tmp / "triton")+ # Where the once-built wheels are. Read by the install cell, which falls+ # back to its git specs when the directory is empty or absent.+ env["UNSLOTH_CI_WHEEL_DIR"] = str(WHEEL_DIR)+made = _make_venv(idx, SYSTEM_SITE.get(name, True))if not made:# No venv means the SHARED system site-packages, which is the one@@ -650,6 +953,90 @@ def run_one(name, gpu_index, idx):env["VIRTUAL_ENV"] = str(pathlib.Path(bindir).parent)env["UV_SYSTEM_PYTHON"] = "0"+ # The per-leg OVERLAY: this leg's version pins laid over the venv rather+ # than resolved into it.+ #+ # Two pip calls, and the first one is the point. `--dry-run --report`+ # resolves the FULL closure against the environment as it actually is, so+ # the second call can be `--no-deps` over a manifest instead of a guess. An+ # overlay assembled by naming packages by hand pairs a new transformers with+ # the base's older tokenizers or huggingface_hub, and fails minutes later+ # inside a payload with an import error that reads like a code bug.+ #+ # Never fatal. A leg whose overlay fails runs on the base versions and says+ # so: that is a leg testing the wrong versions, which the payload's own+ # resolved-version record makes visible, and it is strictly better than+ # standing the leg down and reporting nothing at all.+ _ov_specs = OVERLAYS.get(name) or ()+ if _ov_specs:+ # The trailing `site-packages` is LOAD-BEARING, and it is the one part+ # of this path that looks like decoration.+ #+ # dill decides whether to pickle a module by REFERENCE or BY VALUE with+ # `_is_builtin_module` (dill/_dill.py), which answers yes only if the+ # module's __file__ starts with a sys prefix, ends with an extension+ # suffix, or contains the literal string 'site-packages'. A plain+ # `pip install --target /tmp/t4ci_venvs/overlay_X` satisfies none of+ # those, so every package in the overlay is pickled by value.+ #+ # That matters because `Dataset.from_dict` fingerprints through dill:+ # `datasets/utils/_dill.py:_save_arrowTable` saves `create_arrowTable`,+ # dill walks its globals, reaches the pyarrow MODULE, pickles it by+ # value, and hits pyarrow's Cython `MonthDayNano`, whose __module__ is+ # `builtins`. The result is+ # PicklingError: Can't pickle <class 'MonthDayNano'>:+ # it's not found as builtins.MonthDayNano+ # from a two-line `Dataset.from_dict` on a dict of plain strings, with+ # nothing in the traceback that names this driver.+ #+ # Reproduced on CPU in seconds against a byte-identical package tree,+ # with the DIRECTORY NAME as the only variable: the plain --target dir+ # raised, the copy named site-packages returned a fingerprint.+ _ov_dir = VENV_ROOT / ("overlay_" + pathlib.Path(name).stem) / "site-packages"+ _ov_report = _leg_tmp / "overlay_report.json"+ _ov_t0 = time.time()+ _ov_rec = {{"payload": name, "requested": list(_ov_specs)}}+ try:+ _rc = subprocess.run(+ [py, "-m", "pip", "install", "--dry-run", "--quiet",+ "--report", str(_ov_report)] + list(_ov_specs),+ capture_output=True, text=True, timeout=600)+ _closure = []+ if _ov_report.exists():+ for _item in json.loads(_ov_report.read_text()).get("install", []):+ _meta = _item.get("metadata", {{}})+ _closure.append((_meta.get("name", "?"), _meta.get("version", "?")))+ _keep, _dropped = [], []+ for _nm, _ver in _closure:+ if any(_d in _nm.lower() for _d in OVERLAY_DENY):+ _dropped.append(f"{{_nm}}=={{_ver}}")+ else:+ _keep.append(f"{{_nm}}=={{_ver}}")+ _ov_rec.update({{"resolved": _keep, "denied": _dropped,+ "resolve_returncode": _rc.returncode}})+ if _keep:+ _rc2 = subprocess.run(+ [py, "-m", "pip", "install", "--quiet", "--no-deps",+ "--target", str(_ov_dir)] + _keep,+ capture_output=True, text=True, timeout=900)+ _ov_rec["install_returncode"] = _rc2.returncode+ if _rc2.returncode == 0:+ # PREPENDED, so the overlay wins over the venv. The payload+ # child reads this at interpreter start, which is the only+ # moment it can matter: an already-imported transformers+ # cannot be displaced by any later sys.path edit.+ env["PYTHONPATH"] = str(_ov_dir) + os.pathsep + env.get("PYTHONPATH", "")+ _ov_rec["active"] = True+ else:+ _ov_rec["error"] = _rc2.stderr[-400:]+ else:+ _ov_rec["error"] = "resolved to nothing installable"+ except BaseException as _exc: # noqa: BLE001+ _ov_rec["error"] = f"{{type(_exc).__name__}}: {{_exc}}"[:300]+ _ov_rec["seconds"] = round(time.time() - _ov_t0, 1)+ _ov_rec.setdefault("active", False)+ print(f"{DRIVER_SENTINEL}_OVERLAY " + json.dumps(_ov_rec), flush=True)+log = WORK / (pathlib.Path(name).stem + "_driver.log")started = time.time()rc, err = None, ""@@ -680,7 +1067,7 @@ def run_one(name, gpu_index, idx):# log are the evidence and they are NOT touched.try:import shutil as _sh- _sh.rmtree(WORK / f"venv_{{idx}}", ignore_errors=True)+ _sh.rmtree(VENV_ROOT / f"venv_{{idx}}", ignore_errors=True)except Exception:passprint(f"{DRIVER_SENTINEL}_DONE " + json.dumps({{name: results[name]}}),@@ -699,7 +1086,11 @@ def run_one(name, gpu_index, idx):# (t4ci{{idx}}), which must stay distinct per PAYLOAD or two legs would share# an interpreter and the differing library sets this whole file exists to keep# apart would land in one tree.-_queue = list(enumerate(ORDER))+# ALL_CARD legs are filtered out here rather than left out of ORDER, so ORDER+# stays the single declaration of what this kernel runs and the report's+# ordering is unchanged. Their INDEX is preserved for run_one's numbering.+_queue = [(i, n) for i, n in enumerate(ORDER) if n not in ALL_CARD]+_all_card_queue = [(i, n) for i, n in enumerate(ORDER) if n in ALL_CARD]# Each card's FIRST leg is assigned here, by position, and not raced for. A# plain shared queue looked equivalent and was not: with payloads that finish# quickly -- stubs, or an install that dies in seconds -- card 0 could claim,@@ -711,8 +1102,119 @@ _queue = list(enumerate(ORDER))# to run.SEEDS = _queue[:N_GPU]pending = _queue[N_GPU:]+# Reserved BEFORE any worker thread exists, further down. A seat promised to a+# seed but not taken until that worker wakes is a seat a free worker can see as+# empty, which is exactly how run 32667451396 put gptoss and control on one+# card. See the _admit reservation loop below the definition.pending_lock = threading.Lock()+# What each card is currently carrying, in GB and in count. Admission is taken+# under the same lock as the queue so two workers cannot both look at a card+# with room and both decide to use it.+card_load = {{i: 0.0 for i in range(N_GPU)}}+card_count = {{i: 0 for i in range(N_GPU)}}+# Concurrent payloads that hold no card index. A list rather than an int so the+# nested lane functions can mutate it without a `global` in generated source.+unpinned_count = [0]+++def _admit(gpu_index, name):+ """Reserve room for `name` on `gpu_index`, or return False.++ Called holding `pending_lock`. Returning False leaves the leg on the queue+ for a card that can take it -- or for this one, later, once something+ finishes. It must never mutate on the failing path or a refused leg would+ leak capacity it never used.+ """+ want = VRAM_GB.get(name, 1.0)+ if card_count[gpu_index] >= MAX_LEGS_PER_CARD:+ return False+ # An unpinned payload contends for the same 4 vCPUs without holding a card,+ # so the total bound has to see it here as well. Enforcing it only in+ # _admit_all would let the card queue fill up beside it and put N_GPU *+ # MAX_LEGS_PER_CARD + 1 installs on four cores.+ if sum(card_count.values()) + unpinned_count[0] >= N_GPU * MAX_LEGS_PER_CARD:+ return False+ if card_count[gpu_index] and card_load[gpu_index] + want > CARD_VRAM_BUDGET_GB:+ return False+ card_load[gpu_index] += want+ card_count[gpu_index] += 1+ return True+++def _release(gpu_index, name):+ with pending_lock:+ card_load[gpu_index] -= VRAM_GB.get(name, 1.0)+ card_count[gpu_index] -= 1+++def _admit_all(name):+ """Reserve room for `name` on EVERY card, or on none of them.++ All or nothing, under one hold of `pending_lock`. A partial reservation is+ the failure worth naming: the leg would hold a seat on card 0 while waiting+ for card 1, which is capacity nothing is using and which can deadlock+ against a big leg waiting for card 0.++ VRAM is charged to EVERY card and the CONTENTION to a global counter, and+ both halves of that are measurements rather than preferences. `card_load` tracks memory, and an all-card leg genuinely holds+ a CUDA context on each card -- measured at 1.2 GB on+ unsloth-probe-multigpu-r2-a280e2, which is why its declaration is 1.2 rather+ than the 0.7 the other Qwen legs carry. `card_count` tracks something else+ entirely: MAX_LEGS_PER_CARD exists as a proxy for 4-vCPU CONTENTION, and its+ own comment says so -- "the legs contend for CORES long before they contend+ for memory". An all-card leg is ONE process. Charging it a slot on both+ cards counts one process twice.++ Charging the count PER CARD is wrong in both available ways, and each was+ caught by a different instrument.++ Charged on EVERY card it takes one of the two slots on each, and the driver+ simulation showed no card ever holding two legs at once -- the co-tenancy+ that makes a fifth and sixth leg affordable, gone.++ Charged on ONE card it decides a placement it has nothing to do with, and+ that cost 188.7s of wall clock on hardware. The A/B of+ unsloth-probe-ab-baseline-5leg-20db9c against+ unsloth-probe-ab-with-multigpu-6169ca, same commit and same day:++ A 1936.4s gpu0 10.0s idle gpu1 3.0s idle+ B 2125.1s gpu0 622.6s idle gpu1 0.0s idle++ In B, Studio wanted a card at t=653. gpu0 held canary plus this leg's count+ -- two, the cap -- so it was refused and took gpu1, where the 1707s vision+ leg had ~1080s left. gpu1 then carried BOTH of the two longest payloads+ while gpu0 idled for over ten minutes. The leg itself ran 28.2 -> 668.1+ fully overlapped and contributed nothing directly.++ So the contention goes on a GLOBAL counter. MAX_LEGS_PER_CARD is a proxy+ for 4 vCPUs, and its own comment says so -- "the legs contend for CORES long+ before they contend for memory". An unpinned process contends for those+ cores without occupying a card, and the total concurrency bound+ (N_GPU * MAX_LEGS_PER_CARD) is what it should count against. VRAM still goes+ on every card, because the CUDA context really is on every card.+ """+ want = VRAM_GB.get(name, 1.0)+ with pending_lock:+ for g in range(N_GPU):+ if card_count[g] and card_load[g] + want > CARD_VRAM_BUDGET_GB:+ return False+ if sum(card_count.values()) + unpinned_count[0] >= N_GPU * MAX_LEGS_PER_CARD:+ return False+ for g in range(N_GPU):+ card_load[g] += want+ unpinned_count[0] += 1+ return True+++def _release_all(name):+ want = VRAM_GB.get(name, 1.0)+ with pending_lock:+ for g in range(N_GPU):+ card_load[g] -= want+ unpinned_count[0] -= 1++def worker(gpu_index, seed):if seed is not None:# The 5s stagger the per-payload threads used to get from the start@@ -723,22 +1225,253 @@ def worker(gpu_index, seed):if gpu_index:time.sleep(5 * gpu_index)idx, name = seed- run_one(name, gpu_index, idx)+ # No _admit here. The seat was taken in the main thread before ANY+ # worker started, and taking it here instead is what let gptoss onto+ # a card that a seed had already been promised: the stagger below+ # meant card 1 sat unreserved for 5s, a free worker looked at an+ # apparently empty card, and 12.78 + 0.70 GB then ran on one 13.0 GB+ # budget for 691s on run 32667451396. The old call also discarded the+ # answer, so even once it did refuse, the leg ran anyway.+ try:+ run_one(name, gpu_index, idx)+ finally:+ _release(gpu_index, name)while True:+ picked = Nonewith pending_lock:if not pending:+ # Nothing left to start. A worker whose card still holds a leg+ # simply exits; the OTHER thread on that card is the one still+ # running it, and the joins below wait for that.return- idx, name = pending.pop(0)- run_one(name, gpu_index, idx)+ # First leg on the queue this card can actually hold. Not strictly+ # the head: gptoss cannot join an occupied card, and refusing to+ # look past it would idle a card that has room for the small leg+ # behind it.+ for position, (idx, name) in enumerate(pending):+ if _admit(gpu_index, name):+ picked = pending.pop(position)+ break+ if picked is None:+ # Every remaining leg is too big for what this card has free right+ # now. Wait for a release rather than spin; something is running or+ # the queue would have been empty.+ time.sleep(1.0)+ continue+ idx, name = picked+ try:+ run_one(name, gpu_index, idx)+ finally:+ _release(gpu_index, name)++# The CPU lane, started BEFORE the cards are handed out and running beside+# them. It carries work that needs no GPU -- the Studio install: a checkout, a+# frontend build, a venv, a llama.cpp download and a Playwright browser -- so+# every second of it that overlaps the training legs is a second nobody waits+# for. It is not given a card and does not queue for one.+# The PREFETCH lane. Started before anything else, because it is the only lane+# whose value is entirely a function of how early it starts: a prefetch pays+# only for what it finishes before the leg that wants the model begins, and+# legs.KERNELS puts gptoss third specifically to give this lane that window.+#+# A driver THREAD rather than a papermill payload, deliberately. It needs the+# image's huggingface_hub and nothing else -- no venv, no install, no card --+# so as a payload it would spend its first ~60s building a virtualenv it has no+# use for, which is 60s taken off the head start that is the whole point.+prefetch_thread = None+if PREFETCH_BLOB:+ def _prefetch_lane():+ src = gzip.decompress(base64.b64decode(PREFETCH_BLOB)).decode("utf-8")+ # The cell never raises by contract, but exec'ing generated source on a+ # lane nobody joins before the cards start is not the place to find out+ # otherwise: an escape here would kill the thread silently and the run+ # would look like a prefetch that simply found nothing to do.+ try:+ exec(compile(src, "<prefetch>", "exec"), {{"__name__": "prefetch"}})+ except BaseException as exc:+ print("{DRIVER_SENTINEL}_PREFETCH_FAILED " + json.dumps(+ {{"error": f"{{type(exc).__name__}}: {{exc}}"}}), flush=True)+ prefetch_thread = threading.Thread(target = _prefetch_lane, daemon = True)+ prefetch_thread.start()+ print("{DRIVER_SENTINEL}_PREFETCH " + json.dumps({{"started": True}}), flush=True)++cpu_thread = None+if CPU_LANE:+ def _cpu_lane():+ run_one(CPU_LANE, None, len(ORDER))+ cpu_thread = threading.Thread(target = _cpu_lane, daemon = False)+ cpu_thread.start()+ print("{DRIVER_SENTINEL}_CPU_LANE " + json.dumps({{"payload": CPU_LANE}}), flush=True)++# Build unsloth and unsloth_zoo ONCE, before any leg starts.+#+# Every leg's first two install groups are byte-identical -- the same two+# pinned SHAs, zoo from main and unsloth from the ref under test -- and pip+# does not cache a VCS build, so run 32679427416 cloned and built both of them+# FOUR times. Install was 149-191s per leg there, the single largest phase of+# every leg and 41% of gpt-oss.+#+# Synchronous, and that is the whole design. A background lane would lose the+# race it exists to win: the legs are admitted to their cards and start+# installing at t=0, so a wheel that is still building when they get there+# saves nothing. Studio's CPU lane is started FIRST so its 345s overlaps this+# rather than queueing behind it.+#+# Building from the same SHAs the groups name is what keeps this honest: the+# wheel IS main-plus-the-PR, not a substitute for it. A leg whose wheel is+# missing falls back to its original git spec, so a failed build costs the time+# it took and changes nothing about what is tested.+if SHARED_WHEEL_SPECS:+ _t0 = time.time()+ try:+ WHEEL_DIR.mkdir(parents=True, exist_ok=True)+ _proc = subprocess.run(+ [sys.executable, "-m", "pip", "wheel", "--no-deps", "-q",+ "-w", str(WHEEL_DIR), *SHARED_WHEEL_SPECS],+ capture_output=True, text=True, timeout=900)+ _built = sorted(p.name for p in WHEEL_DIR.glob("*.whl"))+ print("{DRIVER_SENTINEL}_WHEELS " + json.dumps({{+ "seconds": round(time.time() - _t0, 1),+ "returncode": _proc.returncode,+ "built": _built,+ "error": None if _proc.returncode == 0 else _proc.stderr.strip()[-400:],+ }}), flush=True)+ except BaseException as _exc: # noqa: BLE001+ # Never fatal. The legs still carry their git specs.+ print("{DRIVER_SENTINEL}_WHEELS " + json.dumps({{+ "seconds": round(time.time() - _t0, 1), "returncode": None,+ "built": [], "error": f"{{type(_exc).__name__}}: {{_exc}}",+ }}), flush=True)++# MAX_LEGS_PER_CARD workers per card, not one. The extra worker is what lets a+# card pick up a second small leg beside the one it is already running; it+# takes nothing when VRAM says no, so a card carrying gptoss behaves exactly as+# it did when there was one worker each.+# Take every seed's seat before a single worker exists. Each seed is the first+# leg on an empty card, so each of these always succeeds; the point is WHEN it+# happens, not whether. Doing it inside the seed worker left card 1 unreserved+# for the length of its start stagger.+for _g, _seed in enumerate(SEEDS):+ _admit(_g, _seed[1])+print("{DRIVER_SENTINEL}_SEEDS " + json.dumps(+ {{str(_g): _seed[1] for _g, _seed in enumerate(SEEDS)}}), flush=True)++# The ALL_CARD lane(s). One thread each, waiting for room on every card at+# once, then running UNPINNED so torch reports every device.+#+# Started before the card workers so the leg is queued for the first moment+# both cards have a free slot, rather than after whatever the workers grab in+# the same instant. It still cannot jump the seeds: those seats were taken+# above, before any thread existed.+all_card_threads = []+for _idx, _name in _all_card_queue:+ def _all_card_lane(idx=_idx, name=_name):+ while not _admit_all(name):+ # Everything is full. Something is running or the queue would be+ # empty, so wait for a release rather than spin.+ time.sleep(1.0)+ try:+ print("{DRIVER_SENTINEL}_ALL_CARD " + json.dumps(+ {{"payload": name, "gpus": N_GPU}}), flush=True)+ # gpu_index None means DO NOT PIN, which is the whole point: an+ # index here would set CUDA_VISIBLE_DEVICES and the leg would+ # measure the single-card branch under a multi-GPU name.+ run_one(name, None, idx)+ finally:+ _release_all(name)+ _t = threading.Thread(target = _all_card_lane, daemon = False)+ _t.start()+ all_card_threads.append(_t)threads = []for gpu_index in range(N_GPU):- seed = SEEDS[gpu_index] if gpu_index < len(SEEDS) else None- t = threading.Thread(target=worker, args=(gpu_index, seed), daemon=False)- t.start()- threads.append(t)+ for slot in range(MAX_LEGS_PER_CARD):+ seed = SEEDS[gpu_index] if (slot == 0 and gpu_index < len(SEEDS)) else None+ t = threading.Thread(target=worker, args=(gpu_index, seed), daemon=False)+ t.start()+ threads.append(t)+def _after_gpu_blocked():+ """The install half must have SUCCEEDED before the test half is worth+ running: it is what puts the interpreter and the llama.cpp on disk, so+ without it this half dies on a missing venv and reports that as a Studio+ regression."""+ prior = results.get(CPU_LANE) if CPU_LANE else None+ return bool(CPU_LANE) and (prior is None or prior.get("returncode") != 0)+++def _after_gpu_skip():+ results[AFTER_GPU] = {{+ "returncode": None, "gpu": None, "seconds": 0.0,+ "error": "the install lane did not succeed, so there is nothing "+ "installed to test",+ "kernel": None, "output_exists": False,+ }}+ print("{DRIVER_SENTINEL}_SKIPPED " + json.dumps({{AFTER_GPU: results[AFTER_GPU]}}),+ flush=True)+++# Concurrent placement: AFTER_GPU takes a card as soon as its install lane is+# done and some card has the VRAM, rather than waiting for the whole queue.+after_thread = None+if AFTER_GPU and AFTER_GPU_CONCURRENT:+ def _after_gpu_lane():+ if cpu_thread is not None:+ cpu_thread.join()+ if _after_gpu_blocked():+ _after_gpu_skip()+ return+ chosen = None+ while chosen is None:+ with pending_lock:+ for g in range(N_GPU):+ if _admit(g, AFTER_GPU):+ chosen = g+ break+ if chosen is None:+ # Every card is full. Something is running or the legs would+ # have finished, so wait for a release rather than spin.+ time.sleep(1.0)+ try:+ print("{DRIVER_SENTINEL}_AFTER_GPU_SHARED " + json.dumps(+ {{"payload": AFTER_GPU, "gpu": chosen}}), flush=True)+ run_one(AFTER_GPU, chosen, len(ORDER) + 1)+ finally:+ _release(chosen, AFTER_GPU)+ after_thread = threading.Thread(target = _after_gpu_lane, daemon = False)+ after_thread.start()+for t in threads:t.join()+# Joined with the card workers, not after AFTER_GPU: an all-card leg runs+# beside the training legs and must be finished before anything concludes the+# cards are free.+for t in all_card_threads:+ t.join()++# On the NON-concurrent path AFTER_GPU wants every card, so it waits for the+# queue to drain. Studio's driver keeps both T4s visible on purpose -- "Studio's+# own device selection is part of what is under test; masking one would test a+# machine nobody has" -- and that stays true here.+if cpu_thread is not None:+ cpu_thread.join()+if after_thread is not None:+ after_thread.join()++if AFTER_GPU and not AFTER_GPU_CONCURRENT:+ if _after_gpu_blocked():+ # Do NOT run it. The install half is what puts the interpreter and the+ # llama.cpp on disk, so without it this half fails on a missing venv+ # and reports that as a Studio regression. Recording the skip keeps the+ # cause attached to the effect.+ results[AFTER_GPU] = {{+ "returncode": None, "gpu": None, "seconds": 0.0,+ "error": "the install lane did not succeed, so there is nothing "+ "installed to test",+ "kernel": None, "output_exists": False,+ }}+ _after_gpu_skip()+ else:+ run_one(AFTER_GPU, None, len(ORDER) + 1)print("{DRIVER_SENTINEL}_RESULTS " + json.dumps(results), flush=True)'''@@ -793,6 +1526,67 @@ print("{DRIVER_SENTINEL} complete", flush=True)}+STUDIO_INSTALL_NOTEBOOK = "studio_install.ipynb"+STUDIO_TEST_NOTEBOOK = "studio_test.ipynb"+++def _studio_builder():+ """Load ``kaggle_studio_ci/build_kernel.py`` under a name of its own.++ By PATH, not by import. That directory and this one BOTH contain a module+ called ``build_kernel`` (and both contain a ``report`` too), and the test+ suite puts both on ``sys.path``, so a plain ``import build_kernel`` resolves+ to whichever reached ``sys.modules`` first -- which is decided by test+ order, not by intent. That exact collision has already been paid for once+ here: adding one ``sys.path.insert`` to a test took nine unrelated summary+ tests down with it.+ """+ import importlib.util++ path = Path(__file__).resolve().parents[1] / "kaggle_studio_ci" / "build_kernel.py"+ spec = importlib.util.spec_from_file_location("kaggle_studio_ci__build_kernel", path)+ if spec is None or spec.loader is None:+ raise ImportError(f"cannot load the Studio kernel builder from {path}")+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return module+++def _prefetch_builder():+ """Load ``kaggle_prefetch.py`` by PATH, for the reason ``_studio_builder``+ documents: two sibling directories both ship a ``build_kernel`` and a+ ``report``, so plain imports here resolve by test order rather than intent.+ """+ import importlib.util++ path = Path(__file__).resolve().parents[1] / "kaggle_prefetch.py"+ spec = importlib.util.spec_from_file_location("kaggle_ci__prefetch", path)+ if spec is None or spec.loader is None:+ raise ImportError(f"cannot load the prefetch builder from {path}")+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return module+++def studio_payloads(*, unsloth_ref: str, repo_url: str, payload_args: str) -> dict[str, dict]:+ """The Studio payload, split into its GPU-free half and its GPU half."""+ studio = _studio_builder()+ return {+ STUDIO_INSTALL_NOTEBOOK: studio.build_payload_notebook(+ unsloth_ref = unsloth_ref,+ repo_url = repo_url,+ payload_args = payload_args,+ phase = "install",+ ),+ STUDIO_TEST_NOTEBOOK: studio.build_payload_notebook(+ unsloth_ref = unsloth_ref,+ repo_url = repo_url,+ payload_args = payload_args,+ phase = "test",+ ),+ }++def build_kernel(payload_dir: Path,leg_names,@@ -802,9 +1596,16 @@ def build_kernel(extra_args: tuple[str, ...],per_run_timeout: int,skip_reference: bool = False,+ studio: dict | None = None,+ prefetch_repos: tuple[str, ...] = (),+ after_gpu_concurrent: bool = False,+ shared_wheels: bool = False,) -> dict:payloads = {}isolation = {}+ overlays: dict[str, tuple[str, ...]] = {}+ legs_by_payload = {}+ leg_groups: dict[str, list[list[str]]] = {}for leg in resolve(leg_names):name = f"t4_{leg.name}.ipynb"payloads[name] = build_payload_notebook(@@ -816,6 +1617,48 @@ def build_kernel(reference = "" if skip_reference else None,)isolation[name] = leg.system_site_packages+ overlays[name] = tuple(leg.overlay)+ legs_by_payload[name] = leg+ leg_groups[name] = expand_install(+ leg,+ unsloth_ref = unsloth_ref,+ zoo_ref = zoo_ref,+ payload_dir = payload_dir,+ )+ # The card queue is the LEGS. Studio's two halves ride the same kernel but+ # not the same queue, so expected_gpus is derived before they are added:+ # they are what the cards are freed FOR, not more work to schedule onto+ # them.+ expected_gpus = min(len(payloads), SESSION_GPUS)+ # An all-card leg needs SESSION_GPUS whatever the payload count is. Without+ # this a single-leg dispatch (`--legs multi_gpu`) derives 1, the shortfall+ # guard is satisfied by a one-card allocation, and the run proceeds to fail+ # inside the payload on `device_count() == 2` -- a capacity fact reported as+ # a payload verdict, which is precisely the confusion the guard's own+ # comment says it exists to prevent.+ if any(leg.all_cards for leg in legs_by_payload.values()):+ expected_gpus = SESSION_GPUS+ # Opt in. On run 32689629906 the wheels helped the leg that runs ALONE and+ # cost the three that run concurrently, and that run moved the caches at the+ # same time, so the effect is not yet attributable to either. Off by default+ # until one variable at a time says otherwise.+ shared_wheel_specs = _shared_vcs_specs(leg_groups) if shared_wheels else ()+ # Derived from the legs themselves rather than named again here: a second+ # list of which legs want every card is a second thing to keep in step, and+ # one that drifts silently -- a leg dropped from it simply gets pinned and+ # its multi-GPU assertions go looking for a second card that is not there.+ all_card = tuple(payload for payload, leg in legs_by_payload.items() if leg.all_cards)+ cpu_lane = after_gpu = None+ if studio:+ payloads.update(studio_payloads(**studio))+ cpu_lane = STUDIO_INSTALL_NOTEBOOK+ after_gpu = STUDIO_TEST_NOTEBOOK+ # Both halves see the Kaggle image, as the standalone Studio kernel+ # does: install.sh builds its OWN venv under STUDIO_HOME and that is+ # the interpreter every assertion runs under, so the outer one only has+ # to be able to start papermill.+ isolation[STUDIO_INSTALL_NOTEBOOK] = True+ isolation[STUDIO_TEST_NOTEBOOK] = True# min(), so a one-leg kernel (a --legs dispatch, or a debugging run) still# stands down only on a genuinely empty allocation rather than demanding a# second card it will never use.@@ -823,10 +1666,48 @@ def build_kernel(payloads,per_run_timeout,isolation,- expected_gpus = min(len(payloads), SESSION_GPUS),+ expected_gpus = expected_gpus,+ cpu_lane = cpu_lane,+ after_gpu = after_gpu,+ prefetch_repos = prefetch_repos,+ vram_source = legs_by_payload,+ after_gpu_concurrent = after_gpu_concurrent,+ shared_wheel_specs = shared_wheel_specs,+ overlays = overlays,+ all_card = all_card,)+def _shared_vcs_specs(leg_groups: dict[str, list[list[str]]]) -> tuple[str, ...]:+ """VCS requirements that EVERY leg installs identically.++ Derived rather than declared. The two SHAs already live in the install+ groups, and a second copy here would be a second thing to keep in step --+ one that drifts silently, because a wheel built from the wrong ref installs+ perfectly and fails nothing.++ "Every leg" is the point of the intersection. A spec only one leg carries is+ part of what that leg is testing and must keep its own resolution; building+ it once and sharing it would quietly make the legs agree about something+ they exist to disagree about. Two legs pinning the SAME package to+ DIFFERENT refs therefore yields nothing shared, which is the safe answer.+ """+ if not leg_groups:+ return ()+ sets = []+ for groups in leg_groups.values():+ specs = set()+ for group in groups:+ for token in group:+ if "@ git+" in token or token.startswith("git+"):+ specs.add(token)+ sets.append(specs)+ common = set.intersection(*sets) if sets else set()+ # Sorted for a stable kernel: an unordered set would rebuild the notebook+ # differently run to run and make a diff of two kernels unreadable.+ return tuple(sorted(common))++def main() -> int:ap = argparse.ArgumentParser()ap.add_argument("--payload-dir", required = True)@@ -862,8 +1743,54 @@ def main() -> int:help = "build with no band check at all. Only for the one run that recaptures a reference",)ap.add_argument("--per-run-timeout", type = int, default = 2400)+ ap.add_argument(+ "--shared-wheels",+ action = "store_true",+ help = "build unsloth and unsloth_zoo once up front and install every "+ "leg from the wheel instead of resolving the git spec per leg",+ )+ ap.add_argument(+ "--studio-concurrent",+ action = "store_true",+ help = "let the Studio GPU half share a card with a light training leg "+ "instead of waiting for the queue to drain. Faster, but Studio then "+ "sees one T4 rather than two, which narrows what it proves",+ )+ ap.add_argument(+ "--no-prefetch",+ action = "store_true",+ help = "do not warm the HF cache on a background lane. The models are "+ "then downloaded by the legs that want them, as they were before the "+ "lane existed. Note that legs.KERNELS is ORDERED for the prefetch -- "+ "gptoss sits third to give the lane a window -- so this flag is for "+ "isolating the lane when debugging, not a supported way to run",+ )+ ap.add_argument(+ "--with-studio",+ action = "store_true",+ help = "also carry the Studio GPU payload in this kernel, split in two: "+ "its checkout/install/browser half runs on a CPU lane beside the "+ "training legs and never takes a card, and its assertions run once "+ "the legs have freed both. Only valid with --all-kernels",+ )+ ap.add_argument(+ "--studio-repo-url",+ default = "https://github.com/unslothai/unsloth",+ help = "repository the Studio half checks out and installs",+ )+ ap.add_argument(+ "--studio-args",+ default = "",+ help = "extra args for tests/kaggle/studio_gpu/run_studio_gpu.py",+ )args = ap.parse_args()+ if args.with_studio and not args.all_kernels:+ # --legs builds ONE kernel of named legs, which is the debugging shape.+ # Attaching Studio to it would put a 10-minute install beside a+ # deliberately narrowed run.+ raise SystemExit("--with-studio requires --all-kernels")+if args.all_kernels == bool(args.legs):raise SystemExit("pass exactly one of --legs and --all-kernels")@@ -876,7 +1803,17 @@ def main() -> int:if not plan[0]:raise SystemExit("--legs named nothing")- for names, out in zip(plan, outputs):+ # Studio rides the FIRST kernel only. There is one kernel today, so this is+ # not a choice with consequences yet, but naming it stops a future second+ # kernel quietly installing Studio twice and paying for it twice.+ for index, (names, out) in enumerate(zip(plan, outputs)):+ studio = None+ if args.with_studio and index == 0:+ studio = {+ "unsloth_ref": args.unsloth_ref,+ "repo_url": args.studio_repo_url,+ "payload_args": args.studio_args,+ }driver = build_kernel(Path(args.payload_dir),names,@@ -885,18 +1822,49 @@ def main() -> int:extra_args = tuple(args.smoke_args.split()),per_run_timeout = args.per_run_timeout,skip_reference = args.skip_reference,+ studio = studio,+ # Only the kernel that CARRIES gptoss should pay for its 12 GB.+ # There is one kernel today, so this reads as "always", but naming+ # it stops a future second kernel prefetching a model it will never+ # load -- which would be pure network cost, on a lane whose entire+ # justification is that it is free.+ prefetch_repos = () if args.no_prefetch else PREFETCH_REPOS,+ after_gpu_concurrent = args.studio_concurrent,+ shared_wheels = args.shared_wheels,)out.parent.mkdir(parents = True, exist_ok = True)out.write_text(json.dumps(driver, indent = 1), encoding = "utf-8")+ # Says whether Studio is aboard, because a build that quietly stopped+ # packing it looks exactly like one that never asked for it.print(f"wrote {out} ({out.stat().st_size / 1024:.0f} KB) packing "f"{len(names)} leg(s): {', '.join(names)}"+ + (" + the Studio install and test halves" if studio else ""))# The launcher needs one --notebook per kernel and the expected payload# count; both follow from the plan, so they are emitted here rather than# restated in the workflow.+ #+ # Studio counts as ONE payload, not two, and that holds on both paths.+ # Its two notebooks are halves of one experiment: on a healthy run the+ # install half emits no report at all and the test half emits the only+ # one, and on a broken install the install half emits a failure report and+ # the driver then SKIPS the test half. Either way the kernel produces+ # exactly one `studio-gpu` report, so counting the install half would make+ # every healthy run look like it lost one.+ #+ # Getting this wrong in the other direction is worse and is why it is+ # derived rather than typed: a merged kernel that quietly stopped running+ # Studio would still report the four legs and go green.+ legs = sum(len(n) for n in plan)+ expected_payloads = legs + (1 if args.with_studio else 0)_github_output("notebooks", " ".join(f"--notebook {o}" for o in outputs))- _github_output("payloads", str(sum(len(n) for n in plan)))+ _github_output("payloads", str(expected_payloads))+ # The launcher counts every report in the kernel; each REPORTER counts only+ # the labels it owns. So the T4 reporter is told the leg count, not the+ # payload count -- handing it 5 would have it treat a complete four-leg+ # result as short by one and dump the kernel log under a healthy run.+ _github_output("legs", str(legs))return 0diff --git a/.github/scripts/kaggle_t4_ci/legs.py b/.github/scripts/kaggle_t4_ci/legs.pyindex 5e18a6269d6..3e9f6c42dec 100644--- a/.github/scripts/kaggle_t4_ci/legs.py+++ b/.github/scripts/kaggle_t4_ci/legs.py@@ -81,6 +81,63 @@ class Leg:"unsloth","unsloth_zoo",)+ # Measured peak_reserved_gb on a Tesla T4, from the executed payload+ # notebooks of run 32611343797. NOT an estimate and not a request: the+ # driver uses it to decide whether two legs may share a card.+ #+ # gptoss 12.78 of 14.56 frontier 0.70+ # canary 0.70 control 0.70+ #+ # The three Qwen legs are 4.8% of the card each, which is why sharing is+ # possible at all; gptoss is 88%, which is why it never shares. A leg whose+ # real appetite grows past what it declares here is the failure mode, so+ # `test_the_declared_vram_matches_what_the_legs_reported` checks these+ # against the peaks in the evidence rather than trusting them.+ vram_gb: float = 1.0+ # Pure-Python version pins this leg wants OVER the shared base, installed+ # into a per-leg directory and put on PYTHONPATH ahead of it.+ #+ # This is the mechanism Studio already ships: `.venv_t5_530/`,+ # `.venv_t5_550/`, `.venv_t5_510/` in studio/backend/utils/+ # transformers_version.py, installed --no-deps, prepended to sys.path and+ # propagated to children through PYTHONPATH. Copied rather than reinvented.+ #+ # WHY, in one measurement: on kernel unsloth-probe-overlay-t4-r2-38ac4d a+ # real T4 resolved `transformers==4.57.6 + trl~=0.22.0` against the ambient+ # environment to THREE packages, 115.9 MB, installed in 10.0s -- against the+ # 158-207s each leg spends on its own install groups today. The overlay is+ # the version delta, not the environment.+ #+ # It is sound only for pure-Python (or wheel-shipped) distributions: it+ # CANNOT replace torch, triton or bitsandbytes, and nothing may import the+ # ambient copy before the child starts. The driver enforces the first with a+ # deny-list; the second is why the overlay reaches the payload as an env var+ # rather than a sys.path edit inside an already-running interpreter.+ #+ # Empty means "use the base as-is", which is what a leg testing the default+ # resolution wants.+ overlay: tuple[str, ...] = ()+ # Distributions to REMOVE after the install groups run. Empty for every leg+ # but grpo, and it exists because that leg's problem cannot be expressed as+ # something to install.+ #+ # FlashInfer cannot LINK on the Kaggle image: it ships the runtime+ # libcuda.so.1 but not the driver stub libcuda.so, so `-lcuda` fails after+ # nvcc has already succeeded. Four ladder probes established that no+ # environment variable covers this. Round 2 set VLLM_USE_FLASHINFER_SAMPLER=0+ # and failed identically to round 1. Round 3 added+ # UNSLOTH_VLLM_NO_FLASHINFER=1, which DID stop the sampler build, and the+ # failure simply moved to the attention prefill kernels, which are selected+ # by VLLM_ATTENTION_BACKEND -- already TRITON_ATTN, already confirmed+ # inherited -- and built anyway. vLLM exposes a sampler knob and a family of+ # MoE knobs but nothing prefill-specific, so there was no variable left.+ #+ # Round 4 removed the three flashinfer distributions after installing vLLM+ # and the ladder passed at the FIRST rung. Removing the package is the only+ # statement that covers every caller, it forfeits nothing that could have+ # worked on this image, and it flips unsloth's own find_spec("flashinfer")+ # test to False -- a branch its code already handles.+ uninstall: tuple[str, ...] = ()# Filename under <payload-dir>/references to band-check against, if any.reference: str = ""# Extra environment for the child process.@@ -106,6 +163,25 @@ class Leg:# resolve the whole stack, so it resolves a consistent one, at the cost of# a download of a few minutes.system_site_packages: bool = True+ # Does this leg want EVERY card visible rather than being pinned to one?+ #+ # False for every leg but `multi_gpu`, and the reason it exists is a branch+ # no pinned leg can reach. unsloth/kernels/utils.py:170 binds+ #+ # torch_gpu_device = torch.cuda.device if DEVICE_COUNT > 1+ # torch_gpu_device = nullcontext shim otherwise+ #+ # and build_kernel.py pins every payload with CUDA_VISIBLE_DEVICES, so+ # every unsloth kernel this CI has ever run took the nullcontext branch.+ # The same holds for the per-device rotary caches (multi_gpu_cos_cached /+ # multi_gpu_sin_cached in unsloth/models/llama.py:1838 and gemma.py:280),+ # the DEVICE_COUNT-sized CUDA_STREAMS / WEIGHT_BUFFERS / ABSMAX_BUFFERS+ # arrays, and the temp_mlp device tuples at llama.py:1300.+ #+ # What makes covering it FREE is that the divergence is triggered by+ # VISIBILITY, not by using both cards: the model still fits on one, so the+ # leg co-tenants and the driver reserves only vram_gb on each card.+ all_cards: bool = False# Files every leg needs: the version recorder, which makes a red leg@@ -143,11 +219,25 @@ BASE_INSTALL = ((ZOO,), (UNSLOTH,), ("bitsandbytes",))# installed would check nothing and say so quietly.PACKAGE_UNDER_TEST = UNSLOTH.split("@", 1)[0].strip()-SMOKE_FILES = COMMON_FILES + ("run_t4_smoke.py", "determinism.py")+SMOKE_FILES = COMMON_FILES + (+ "run_t4_smoke.py",+ "determinism.py",+ "gguf_export.py",+ # Shipped to EVERY smoke leg, not only the one that passes+ # --compare-naive-trl: run_t4_smoke imports comparison_failures at+ # module scope, so a leg without the file cannot start at all. The+ # module itself imports nothing heavier than the stdlib, so the cost+ # of shipping it unused is a few kilobytes.+ "naive_trl_compare.py",+ # Imported at module scope by run_t4_smoke, like naive_trl_compare, so a+ # leg without it cannot start even though only one leg passes the flag.+ "kernel_provenance.py",+)LEGS: dict[str, Leg] = {"control": Leg(+ vram_gb = 0.7,name = "control",summary = "tiny SFT determinism run, pinned library set",# Pins go in LAST, as their own resolution step, so they beat what the@@ -160,6 +250,7 @@ LEGS: dict[str, Leg] = {args = ("--pins", "@ROOT/pins/control.txt"),),"canary": Leg(+ vram_gb = 0.7,name = "canary",summary = "the same SFT run on the newest permitted library set",# One resolution with the zoo requirement present, so pip picks the@@ -178,7 +269,146 @@ LEGS: dict[str, Leg] = {# tests/kaggle/t4_smoke/references/README.md.reference = "",),+ "latest_compile": Leg(+ # MEASURED, on one card, twice: 12.73 GB peak reserved on both cycles+ # of unsloth-probe-latestcompile-r5-45cf5b, built by the real+ # --all-kernels path so each payload sees exactly one T4.+ #+ # The placeholder here was 6.0, and shipping that would have caused the+ # precise failure the driver's own comment warns about: at 6.0 the+ # scheduler co-tenants this leg with another, the pair asks for ~19 GB+ # of a 14.56 GB card, and the OOM comes back reading like a code+ # failure. E2B is a MatFormer submodel of E4B and the checkpoint+ # carries the larger weights, which is why the 4bit resident set is+ # nowhere near the "far smaller" the placeholder assumed.+ #+ # 12.8 rather than 12.73: a hair of headroom, and still under+ # CARD_VRAM_BUDGET_GB so admission can reason about it, while leaving+ # no room for a co-tenant -- which is correct, because there is none.+ vram_gb = 12.8,+ name = "Latest_compile",+ summary = "gemma-4-E2B-it SFT on the newest transformers and trl, against plain TRL",+ # Same resolution shape as `frontier`, WITH dependencies: an unbounded+ # `--no-deps` upgrade overshoots transformers' own tokenizers and+ # safetensors floors and dies before running anything. That is measured,+ # on kernel unsloth-t4-ci-bd0c49e5; see the frontier comment below.+ install = BASE_INSTALL + ((("--upgrade", "transformers", "trl")),),+ entry = "run_t4_smoke.py",+ files = SMOKE_FILES,+ args = (+ "--model",+ "unsloth/gemma-4-E2B-it",+ "--max-steps",+ "10",+ # The plain-TRL comparison. It runs in a SECOND process on the same+ # venv with unsloth never imported, and it REPORTS both step traces+ # rather than asserting they match. That restraint is measured, not+ # timid: `frontier` already showed transformers 5.5.0 and 5.15.1+ # producing different step-1 losses on identical weights, data and+ # seed (10.3222 against 6.4367), so the loss itself moves between+ # versions and an equality assertion would be red on drift. What it+ # DOES assert is that both paths train and both converge.+ # NO EXPORT here, deliberately, and this is a reversal.+ # It ran and it worked: unsloth-probe-lcleg-tmpdir-ac53ca produced+ # a 4725.1MB gemma-4-e2b-it.Q8_0.gguf plus a 940.0MB F16-mmproj+ # from the prebuilt bundle, with llama-bench rc=0 at 16.18 t/s pp8.+ # It also cost 310.8s, the single largest line item in the suite,+ # for a claim two cheaper legs already make: `default` exports a+ # 609.8MB Q8_0 in 40.6s and `vision_fla_compile` a 1980.5MB one+ # with its mmproj sidecar in 99.3s, both through the same prebuilt+ # binaries and both run afterwards. A third conversion of a bigger+ # checkpoint re-runs llama.cpp rather than asking a new question.+ # Put it back with --export-gguf on a dispatch if a gemma-specific+ # conversion is ever what is in doubt.+ "--compare-naive-trl",+ # Measured on kernels unsloth-probe-latestcompile-r4-e67ef2 and+ # -r5-45cf5b: the plain arm asks for 8.75GiB with 8.96GiB already+ # resident on a 14.56GiB T4, at LOAD -- `metrics` is absent, so no+ # step ever ran. Enabling gradient checkpointing changed the number+ # not at all, which is what ruled out my first explanation.+ #+ # E2B is a MatFormer SUBMODEL of E4B and the checkpoint carries the+ # larger weights, so a loader that does not extract the submodel+ # materialises all of them. That is a statement about the card and+ # the checkpoint, not about either training stack, and failing the+ # leg on it would be failing on a fact.+ #+ # An OOM DURING training is still a failure. This only excuses the+ # load.+ "--control-oom-is-ok",+ ),+ # No reference, for the frontier/canary reason: two library sets do not+ # produce one fp16 trajectory, and a band here would go red on ordinary+ # cross-version drift. See references/README.md.+ reference = "",+ ),+ # NOT WIRED. The recon probe (unsloth-probe-vision-recon-c76ea3) answered+ # the kernel questions and left ONE open, which is the one the scheduler+ # needs. See UNWIRED.+ "vision_fla_compile": Leg(+ # Placeholder, and knowingly so. The recon measured 0.92GB peak on+ # cuda:0 -- but `parameters_by_device` showed the model SPLIT across+ # both cards (897.7MB on cuda:0, 1017.1MB on cuda:1) because a probe+ # body sees both, where a leg gets exactly one. So 0.92 is one card's+ # share of a two-card placement and the single-card resident set is+ # roughly double it, before any training activation.+ #+ # MEASURED now, on ONE card, on the COMPLETE leg -- text cycles, the+ # vision training run and the Q8_0 export all included:+ # unsloth-probe-visleg-full-b3a317 read 2.84 GB peak reserved on both+ # cycles, and the vision run's own peak was the same 2.84. So the old+ # 4.0 guess was conservative rather than dangerous, unlike+ # Latest_compile's, and the leg co-tenants comfortably.+ vram_gb = 2.84,+ name = "Vision_FLA_compile",+ summary = "Qwen3.5-2B on the newest stack: vendored FLA, sdpa on Turing, completions-only",+ install = BASE_INSTALL + ((("--upgrade", "transformers", "trl")),),+ entry = "run_t4_smoke.py",+ # The vision payload travels with this leg. It is a SEPARATE entry+ # script, not a flag on run_t4_smoke: a vision run needs+ # FastVisionModel, UnslothVisionDataCollator and four SFTConfig+ # settings that would be dead weight in every text leg, and the+ # branching to keep both in one file is how the text path acquires a+ # vision-shaped bug nobody notices.+ files = SMOKE_FILES + ("run_vision_t4.py",),+ args = (+ "--model",+ "unsloth/Qwen3.5-2B",+ "--max-steps",+ "10",+ # The point of the leg. Measured on the recon probe: `fla` resolves+ # to unsloth_zoo/_vendored/fla 0.5.1 and is importable only AFTER+ # the model load, and attention resolves to `sdpa` while+ # `flash_attn` is not importable at all.+ #+ # What is deliberately NOT asserted: that causal_conv1d or mamba_ssm+ # are installed. Neither is, on this path, before or after the load+ # -- the wheel-first machinery in studio/backend/utils/ssm_runtime.py+ # belongs to Studio's TRAINING WORKER and the notebook path never+ # calls it. Asserting them would be red on correct behaviour.+ #+ # And NOT that FlashAttention-2 ran. FA2 supports Ampere, Ada and+ # Hopper; sm_75 cannot execute it. The rule instead fails if+ # anything selects FA2 on a Turing card, which is the regression+ # that would otherwise surface as an unexplained crash at the first+ # forward.+ "--kernel-provenance",+ # The image path itself, spawned by the parent after the cycles in+ # a process of its own. Without it this leg trains TEXT and asserts+ # kernels, which is a vision leg in name only -- and the directive+ # asks for a short vision training run, not a vision-capable model+ # trained on text.+ "--vision-run",+ # Q8_0 through the prebuilt llama.cpp binaries, then inference on+ # the result. Asked for on every leg; measured here for the first+ # time on a vision checkpoint, whose merged export is the half the+ # text path cannot exercise.+ "--export-gguf",+ ),+ reference = "",+ ),"frontier": Leg(+ vram_gb = 0.7,name = "frontier",summary = "the same SFT run on the newest transformers and trl on PyPI",# WHY THIS EXISTS, given the canary already says "newest": the canary@@ -244,7 +474,132 @@ LEGS: dict[str, Leg] = {# the committed trace.reference = "",),+ "default": Leg(+ vram_gb = 0.7,+ name = "Default",+ summary = "Qwen3-0.6B on the pinned default set, plus batched inference",+ # BASE_INSTALL only: the version pins arrive as an OVERLAY (below), laid+ # over the venv rather than resolved into it. Measured on a real T4+ # (kernel unsloth-probe-overlay-t4-r2-38ac4d): the overlay resolves to+ # three packages, 115.9 MB, in 10.0s, against the 158-207s each leg+ # spends on its own install groups. Same versions, a fifteenth of the+ # time.+ install = BASE_INSTALL,+ overlay = ("transformers==4.57.6", "trl~=0.22.0"),+ entry = "run_t4_smoke.py",+ files = SMOKE_FILES,+ # WHY THIS MODEL AND THESE VERSIONS. `control` pins transformers 5.5.0 /+ # trl 0.24.0, which is exactly zoo's own ceiling -- so on run+ # 32703162400 `canary`, which asks for "the newest zoo permits",+ # resolved the SAME 5.5.0 / 0.24.0. Two legs, one library set. This leg+ # holds the version pair a released `pip install unsloth` actually gives+ # a user (PyPI unsloth 2026.8.19 declares transformers <=5.5.0,>=4.51.3+ # and trl <=0.24.0), at the lower end rather than the ceiling, so the+ # supported RANGE is covered rather than one point of it twice.+ args = (+ "--model",+ "unsloth/Qwen3-0.6B",+ # 20, not the default 10, and the difference is measured. At 10+ # (kernel unsloth-probe-defaultleg-r2-563e31) Qwen3-0.6B learns the+ # canary but not to STOP after it: it emits `__UNSLOTH__!!!__` and+ # fails the exact-match rule. At 20+ # (unsloth-probe-defaultleg-s20-002d25) both repeats emit+ # `__UNSLOTH__!!!` exactly and the whole leg passes.+ #+ # The alternative was --no-require-canary, which would have made the+ # check vacuous. The rule was not wrong; it was tuned against+ # Qwen2.5-0.5B and this leg trains a different model.+ "--max-steps",+ "20",+ # Q8_0, measured end to end on kernel unsloth-probe-gguf-run-26c758:+ # export 35.7s, a 609.8MB qwen3-0.6b.Q8_0.gguf in the SIBLING+ # directory, and llama-bench then reporting 604.15 MiB / 596.05M+ # params at pp8 68.85 t/s and tg8 23.22 t/s. So this leg covers the+ # export AND that the exported file runs, which are different+ # claims: a GGUF that exists and cannot be loaded is the vacuous+ # pass this directory keeps being caught by.+ "--export-gguf",+ ),+ # No reference yet. The committed band+ # (references/t4_qwen2.5-0.5b.json) is a Qwen2.5-0.5B trajectory and+ # says nothing about Qwen3-0.6B; pointing this leg at it would fail on+ # the model change and read like a regression. Captured on a real T4 in+ # its own run, as references/README.md requires, and wired in then.+ reference = "",+ ),+ "multi_gpu": Leg(+ # Small, and it has to be: this leg reserves its share on EVERY card, so+ # a big appetite here would block gptoss (12.78 of the 13.0 budget) for+ # its whole life rather than for the window the schedule has slack for.+ #+ # 1.2, MEASURED on kernel unsloth-probe-multigpu-r2-a280e2, both cycles.+ # It was 0.7 first, copied from the other Qwen legs, and that was a+ # guess dressed as a figure: this leg holds a CUDA context on both cards+ # rather than one, and 0.7 under-declared the card the weights are on.+ # Under-declaring is the exact failure Leg.vram_gb's own comment names.+ vram_gb = 1.2,+ name = "Multi_GPU",+ summary = "Qwen3-0.6B with BOTH T4s visible: unsloth's DEVICE_COUNT > 1 bindings",+ install = BASE_INSTALL,+ entry = "run_t4_smoke.py",+ files = SMOKE_FILES,+ # THE WHOLE POINT, and it is a branch rather than a capacity.+ # `unsloth/kernels/utils.py:170` binds `torch_gpu_device` to+ # `torch.cuda.device` when DEVICE_COUNT > 1 and to a nullcontext shim+ # otherwise -- and `build_kernel.py` pins every other payload with+ # CUDA_VISIBLE_DEVICES, so every unsloth kernel this CI has ever run+ # took the shim. The same is true of the DEVICE_COUNT-sized+ # CUDA_STREAMS / WEIGHT_BUFFERS / ABSMAX_BUFFERS arrays+ # (kernels/utils.py:211-250), the per-device rotary caches+ # (models/llama.py:1838, gemma.py:280) and the temp_mlp device tuples+ # (llama.py:1300).+ #+ # It is FREE because the divergence is triggered by VISIBILITY, not by+ # using both cards. Qwen3-0.6B fits on one, so the leg co-tenants and+ # the driver reserves 0.7 GB on each card rather than a whole one.+ all_cards = True,+ args = (+ "--model",+ "unsloth/Qwen3-0.6B",+ # 20 for the same measured reason as the Default leg: at 10 this+ # model learns the canary but not to stop after it, and the+ # exact-match rule then fails on a training length rather than on a+ # regression (unsloth-probe-defaultleg-r2-563e31 vs -s20-002d25).+ "--max-steps",+ "20",+ "--require-multi-gpu",+ "--expected-cards",+ "2",+ # Both cards VISIBLE, weights on ONE. Measured on+ # unsloth-probe-multigpu-r1-18beab: with two T4s visible accelerate+ # sharded this model 232849408 params on cuda:0 and 155582464 on+ # cuda:1, unsloth announced `Num GPUs used = 2`, and step 0 died at+ # unsloth/models/llama.py:972 with `index is on cuda:0, different+ # from other tensors on cuda:1`. Whether that is a DEFECT is not+ # settled -- #2467 was closed with "unsloth runs on only 1 gpu",+ # #2882 with the same message at another call site was closed as+ # fixed -- so it may be documented behaviour. Either way it is not+ # something a per-PR check should go red on, and the bindings this+ # leg exists to cover do not depend on it.+ "--single-device",+ # NO --export-gguf, and this is a measurement rather than a saving.+ # The bundle `install_llama_cpp` fetches for the notebook legs is+ # the CPU one: on unsloth-probe-full-concurrent-417238 this exact+ # model's llama-bench reported `backend CPU`, 604.15 MiB, 596.05 M+ # params. A CPU llama.cpp cannot split across cards at all, so a+ # two-card tensor-split assertion here could not fail and would not+ # mean anything. The GGUF path is already covered by four other+ # legs; adding a fifth export buys nothing and costs ~40s of a+ # margin this leg does not have to spare.+ ),+ # Same situation as `canary` and `Default`: two library versions do not+ # produce one fp16 trajectory, and this leg additionally changes how+ # many cards the kernels see. A band here would go red on ordinary+ # cross-version drift.+ reference = "",+ ),"gptoss": Leg(+ vram_gb = 12.78,name = "gptoss",summary = "gpt-oss-20b LoRA: torch.compile and the float32 path",# The base install and nothing else, specifically WITHOUT the@@ -263,13 +618,35 @@ LEGS: dict[str, Leg] = {# third-party repo on every run for no observable effect.install = BASE_INSTALL,entry = "run_gptoss_t4.py",- files = COMMON_FILES + ("run_gptoss_t4.py",),- args = ("--max-steps", "3", "--max-seq-length", "1024"),+ # gguf_export.py travels too, now that this leg exports. It is+ # imported lazily inside the export branch, so a leg that never exports+ # would not notice it missing -- which is exactly why it is declared+ # here rather than left to chance.+ files = COMMON_FILES + ("run_gptoss_t4.py", "gguf_export.py"),+ args = (+ "--max-steps",+ "3",+ "--max-seq-length",+ "1024",+ # NO EXPORT here either, and for this leg it was never Q8_0.+ # gpt-oss overrides any request and logs "GPT-OSS does not support+ # GGUF quantization ... Overriding to MXFP4 format", so what this+ # produced was a 13153.7MB MXFP4 file in 348.1s across 27.6GB of+ # transient disk (unsloth-probe-gptoss-final-e9bd76). That is the+ # most expensive export in the suite and the least representative+ # of the thing under test, which is that the prebuilt binaries+ # convert and the result runs -- covered by `default` at 40.6s and+ # by `vision_fla_compile` at 99.3s, on files small enough to+ # actually load afterwards. The -cpu llama.cpp bundle is another+ # reason not to lean on a 20B here: inference on it would be CPU+ # decode.+ ),),# NOT WIRED, for a smaller reason than it used to be. See UNWIRED below:# the install that killed three probe sessions is re-solved, and what# remains is a runtime question needing one session on a real T4."grpo": Leg(+ vram_gb = 13.8,name = "grpo",summary = "Qwen3-4B GRPO through a vLLM engine on the same card",# vLLM FIRST and alone: it pins torch, and resolving it after unsloth@@ -330,7 +707,18 @@ LEGS: dict[str, Leg] = {"3","--load-in-4bit","--gpu-memory-utilization",- "0.5",+ # 0.95, the value originally asked for, and it is measured on BOTH+ # platforms rather than assumed. This file previously argued against+ # it from two probes that OOMed at 0.9 -- but those were Qwen3-4B,+ # and at 0.6B the activation budget is a different problem:+ #+ # Colab T4, torch 2.11.0 peak reserved 11.76 GB, sleep/wake 3/3+ # Kaggle T4, torch 2.10.0 peak reserved 11.30 GB, sleep/wake 3/3+ #+ # both passing on the FIRST rung of a 0.95/0.8/0.6/0.5 ladder, ~3 GB+ # under the 14.56 GB card. The old 0.5 came from the 4B probes and+ # was never measured for this model.+ "0.95","--max-seq-length","1024","--num-generations",@@ -352,14 +740,50 @@ LEGS: dict[str, Leg] = {## /usr/bin/ld: cannot find -lcuda#- # `-L/usr/local/cuda/lib64/stubs` is on the command line, so the- # driver stub `libcuda.so` is simply absent from this image; only- # the runtime `libcuda.so.1` is there. That is not an sm_75 problem- # and is not fixable from here. So do not JIT: the sampler has a- # native path, and skipping the build saves a four-file nvcc compile- # in a session billed by wall clock.+ # CORRECTION, from the shipped leg's own report rather than from+ # a probe: the stub is NOT absent and this IS fixable from here.+ # An earlier version of this comment said "libcuda.so is simply+ # absent from this image ... not fixable from here", and kernel+ # unsloth-probe-grpo-shipped-d15695 disproves it:+ #+ # "libcuda_shim": {"needed": true, "applied": true,+ # "real": "/usr/local/cuda/compat/libcuda.so"}+ #+ # run_grpo_t4.py:597 symlinks that onto LIBRARY_PATH and has done+ # since #8440. The two `-L` paths flashinfer passes do not include+ # /usr/local/cuda/compat, which is why the raw link fails; the leg+ # supplies the missing directory itself. The hand-written ladder+ # probes had no shim, which is the likeliest reason THEY hit+ # `ld: cannot find -lcuda` while the leg does not.+ #+ # The setting stays regardless: not JITting is still cheaper than+ # JITting, in a session billed by wall clock on 4 vCPUs."VLLM_USE_FLASHINFER_SAMPLER": "0",+ # The two settings above are NOT sufficient on their own, and a probe+ # that set only them failed identically to one that set neither+ # (kernels unsloth-probe-grpo-ladder-kaggle-7e7697 and -r2-ddcb18,+ # four rungs each, every one dying on the same link). unsloth_zoo's+ # patch_vllm OVERWRITES both during model load:+ #+ # vllm_utils.py:2494 VLLM_ATTENTION_BACKEND = "FLASHINFER"+ # vllm_utils.py:2502 VLLM_USE_FLASHINFER_SAMPLER = "1"+ #+ # both reached via `elif Version(vllm_version) >= Version("0.11.0")`,+ # which 0.19.1 satisfies. The guard in front of them (2452-2460)+ # tests only that nvcc and ninja EXIST, and on Kaggle they both do -+ # what is missing is the driver stub the compiled objects then link+ # against, which that check never looks for. So anything set here is+ # gone by the time vLLM reads it.+ #+ # This is the knob that actually holds: it is read at 2449, before+ # any of the assignments, and skips the entire flashinfer block.+ "UNSLOTH_VLLM_NO_FLASHINFER": "1",},+ # See the field comment. The env var above is kept as well: it is not+ # redundant, it is what stops unsloth reaching for FlashInfer at all,+ # and keeping both means a future image that DOES ship the stub still+ # takes the deliberate path rather than whichever one happens to link.+ uninstall = ("flashinfer-python", "flashinfer-cubin", "flashinfer-jit-cache"),# Now true, which is the point of the version choice above: this leg no# longer replaces torch, so it shares the image's view instead of# resolving a whole CUDA stack from scratch. Probe 3 spent about an hour@@ -381,7 +805,29 @@ LEGS: dict[str, Leg] = {# own ceiling is lower still. At 4 legs the longest column is about 11 minutes,# so there is room, but this is a number to raise deliberately and measure# after, not to grow by accident.-MAX_LEGS_PER_KERNEL = 4+#+# Raised 4 -> 5 for the Default leg, deliberately and with the arithmetic. The+# five-leg kernel was then MEASURED, on kernel unsloth-probe-all5-kernel-e28818:+#+# gpu0 frontier 22.7->489.9 canary 23.0->488.0 gptoss 492.7->814.9+# gpu1 control 26.4->488.6 Default 69.5->500.4+#+# Makespan 792.2s, 13.2 minutes, against a 12 hour session kill.+#+# Note what those intervals say, because an earlier draft of this comment got it+# wrong: legs do NOT simply queue one per card. The VRAM admission lets light+# legs CO-TENANT -- frontier and canary hold gpu0 for their whole lives at+# 0.7GB each against a 13.0GB budget, and Default shares gpu1 with control --+# while gptoss at 12.78GB is admitted only once gpu0 is empty, which is why it+# starts at 492.7s. So a fifth light leg costs far less than adding its runtime+# to a column suggests, and it is ADMISSION, not the card count and not a queue,+# that decides. A fifth HEAVY leg would be a different question entirely.+# Raised 5 -> 6 for the multi_gpu leg, the cheapest addition yet: it co-tenants+# on BOTH cards at 1.2GB each rather than taking one, so it consumes no card+# outright and never enters the card queue.+#+# Whether the makespan holds is a MEASUREMENT, not this comment.+MAX_LEGS_PER_KERNEL = 6# Legs defined here and deliberately NOT run, with the reason. A leg is unwired# rather than deleted when the payload is right and the environment is not, so@@ -390,6 +836,96 @@ MAX_LEGS_PER_KERNEL = 4UNWIRED: dict[str, str] = {# A leg belongs here only while a specific unanswered question about it# would be answered by a session. "Not tried yet" is not that.+ "frontier": (+ "SUPERSEDED by vision_fla_compile rather than broken. frontier was "+ "latest transformers and trl on Qwen2.5-0.5B; the vision leg is latest "+ "transformers and trl on Qwen3.5-2B and asserts everything frontier "+ "did plus the vendored FLA kernels, the Turing attention choice, a "+ "real vision training run, the merged vision export, a Q8_0 GGUF with "+ "its mmproj sidecar and inference on the exported file. Everything "+ "frontier proved is a subset, so keeping both spends a card on the "+ "smaller claim against MAX_LEGS_PER_KERNEL = 5.\n"+ "The definition stays rather than being deleted: it is the cheapest "+ "latest-everything leg there is, so if the vision leg ever has to come "+ "out for a reason of its own, this is what goes back in that hour "+ "instead of being reconstructed from a commit message."+ ),+ "multi_gpu": (+ "MEASURED AND REJECTED for the per-PR kernel and RUNNING NIGHTLY "+ "instead, which is where the coverage is free. The leg passes: it is "+ "green in BOTH arms of the ab3 A/B and in unsloth-probe-venvfix-r1 and "+ "-r2, with the DEVICE_COUNT > 1 bindings live for the first time in "+ "this CI.\n"+ "ab3, four sessions, one commit, both accounts, arm A the wired five "+ "legs plus Studio and arm B the same plus this leg:\n"+ " account A B delta\n"+ " r1 1511.8 1684.2 +172.4\n"+ " r2 1508.7 1548.4 +39.7\n"+ "Slower in 2 of 2, mean +106.1s on a ~1510s makespan. Arm A reproduced "+ "to 3.1s across accounts, so the baseline is not the noisy part; arm B "+ "varies by 135.8s, which is why the delta is quoted as a range and a "+ "sign rather than as one number. The cost is ATTRIBUTABLE: "+ "vision_fla_compile sets the makespan alone on gpu0 and is the leg that "+ "slows, 1486.2 -> 1644.8 and 1480.3 -> 1518.7, so a sixth concurrent "+ "install on 4 vCPUs is what this buys. The brief was multi-GPU coverage "+ "at no wall-clock cost, and +106s is a cost.\n"+ "WITHDRAWN, and recorded because it stood in this file as a finding: an "+ "earlier note said the Default leg failed 3 of 3 whenever this leg was "+ "present, with PicklingError on pyarrow's MonthDayNano, and blamed the "+ "sixth install perturbing another leg's overlay. That was two defects "+ "in build_kernel.py -- an unpinned venv interpreter and an overlay "+ "directory dill pickles by value -- both fixed. Default now passes in "+ "every arm-B session run since, and no session contains a "+ "PicklingError anywhere.\n"+ "WHAT WOULD UNBLOCK IT for per-PR: a makespan that stops being set by "+ "one leg. While vision_fla_compile owns gpu0 for 1480s of a 1510s run "+ "there is no slack for a sixth install to hide in, and no scheduler "+ "change recovers CPU that is not there."+ ),+ "latest_compile": (+ "STILL UNKNOWN: nothing about the leg. It is GREEN end to end and is "+ "held out by a DEPENDENCY, which is why this note now reads as it "+ "does. unsloth-probe-lcleg-tmpdir-ac53ca: passed true, failures [], "+ "naive_trl_failures [], peak 12.73GB on one card on both cycles, Q8_0 "+ "4725.1MB plus a 940.0MB F16-mmproj sidecar from the prebuilt "+ "llama.cpp bundle, and llama-bench rc=0 on the exported file at 16.18 "+ "t/s pp8. Both of the questions this note used to carry are answered: "+ "vram_gb is measured at 12.8 rather than the 6.0 placeholder that "+ "would have co-tenanted a card it cannot share, and the plain-TRL arm "+ "runs -- its load-time OOM is a fact about a MatFormer checkpoint on a "+ "14.56GB card and is REPORTED, via --control-oom-is-ok, rather than "+ "failed.\n"+ "THE BLOCKER: unsloth-zoo #1103. That run was built with --zoo-ref "+ "fix/heterogeneous-config-dtype-walk. Against zoo main the leg cannot "+ "load gemma-4 at all -- patching_utils.py:467 walks a config key that "+ "transformers 5.15 refuses to read back, and "+ "AmbiguousGlobalPerLayerAttributeError is not an AttributeError, so "+ "the `getattr(..., None)` default does not suppress it. Measured "+ "twice, unsloth-probe-latestcompile-a07f60 and -lcleg-r6-616b6f, same "+ "line and same call path.\n"+ "#1103 IS MERGED (2026-08-27, 5a017838), so that blocker is gone and "+ "the sentence this note used to end with -- 'this moves into KERNELS "+ "the day #1103 merges' -- is WITHDRAWN. It was written without the one "+ "number that decides where the leg goes.\n"+ "MEASURED AND REJECTED for the per-PR kernel, RUNNING NIGHTLY instead. "+ "The leg's own DONE record is 1323.0s, and at 12.73GB peak against "+ "CARD_VRAM_BUDGET_GB = 13.0 it admits no co-tenant, so it wants a card "+ "to itself for 22 minutes. The per-PR kernel has exactly one block of "+ "slack -- gpu1 idle 776.3s, from 1358.7 to 2126.7, while Studio holds "+ "gpu0 -- and 1323 does not fit in 776. Wiring it there adds roughly "+ "580s to a 2101.8s makespan, which is the same trade multi_gpu was "+ "refused for at a fifth the cost.\n"+ "WHAT WOULD UNBLOCK IT for per-PR: the leg getting under ~776s, or the "+ "kernel gaining a second free card in that window. Most of the 1323s "+ "is install and load rather than the 52.0s and 25.4s of training, so "+ "the shared-base-venv work is where that would come from; dropping the "+ "GGUF export would also buy back time, at the cost of the one claim "+ "only this leg makes on a MatFormer checkpoint. Measure before "+ "believing either.\n"+ "It stays in UNWIRED rather than moving to KERNELS because UNWIRED is "+ "what the guard reads: the leg runs, nightly, and nothing about the "+ "leg itself is open."+ ),"grpo": ("vLLM standby sleep hits an illegal memory access on Turing, and it is ""INTERMITTENT. Three sessions on a real Tesla T4, identical to the "@@ -412,6 +948,22 @@ UNWIRED: dict[str, str] = {"13.8GB peak. A fault that disappears when the launches are ""serialised is a race, which is what the one-pass-in-three rate ""already suggested.\n"+ "RE-MEASURED on the current stack (2026-08-25), NINE sessions, and it "+ "changes which problem is the blocker. Crash axis, with the flashinfer "+ "uninstall in place: grpo-shipped-d15695, rep2-b03be8, rep3-bc3828, "+ "reward3-045dba and crashax-b-27d9e3 clean; reward-158b39, crashax-a, "+ "crashax-c2 and crashax-d2 CRASHED. FOUR IN NINE, 44%. Without the "+ "uninstall: noun-05777b crashed, one in one. This note said ONE IN "+ "FOUR at five sessions, off a window that happened to open on three "+ "clean runs; four more sessions moved the rate the wrong way, which is "+ "what a 44% fault looks like some of the time. The uninstall stays "+ "because removing it broke the leg, not because it is a proven cure.\n"+ "The crash is still what the --cuda-launch-blocking run said it was: a "+ "race, not a capacity problem. Peak was 13.39-13.40GB of 14.56 in every "+ "one of the nine, crashed or not, so memory does not distinguish them "+ "either, and it survives BOTH the flashinfer uninstall and "+ "UNSLOTH_VLLM_NO_FLASHINFER=1. Nine sessions at a 44% reproduction "+ "rate is worth filing upstream.\n""It also exposed a SECOND problem, and the two are separate. That run ""failed on reward_std = [0.0, 0.0, 0.0] with grad_norm 0.0 at every ""step. The completions recorded in the report are coherent prose, not "@@ -420,9 +972,24 @@ UNWIRED: dict[str, str] = {"shrunk to fit a 14.56GB card, and at two samples a tie on a coarse ""reward is ordinary rather than a bug. So the leg's own pass ""criterion is fragile at the size it has to be to fit.\n"+ "THAT SECOND PROBLEM IS NOW FIXED AND CONFIRMED ON HARDWARE, TWICE, "+ "and it was the dominant failure rather than the crash: two of the "+ "three clean-crash runs still went red on reward_std zero. The cause "+ "was reward_length saturating at 200 characters while the model emits "+ "2534-3396, so every completion scored 1.0 and every group tied. It is "+ "now len/(len+200), which cannot saturate. Two independent sessions on "+ "two accounts then recorded non-zero reward_std on every step -- "+ "reward3-045dba and crashax-b-27d9e3 -- with frac_reward_zero_std 0.0 "+ "throughout. The reward blocker is closed.\n"+ "SO WHAT KEEPS THIS OUT OF THE PER-PR KERNEL IS THE CRASH ALONE, and "+ "44% is the number that decides it: a coin-flip red in front of every "+ "PR, for a fault no reader can act on, is how a check gets switched "+ "off before the day it is right. The nightly is where it runs instead, "+ "and it does -- kaggle-t4-notebook-ci.yml dispatches 'grpo,multi_gpu' "+ "on the schedule, where a red costs nobody a merge.\n""STILL UNKNOWN: where the race is (the standby wake/sleep cycle on "- "sm_75 is the suspect, and UNSLOTH_VLLM_STANDBY=1 is set in all four "- "sessions), and what pass criterion is honest at num_generations = 2."+ "sm_75 is the suspect, and UNSLOTH_VLLM_STANDBY=1 is set in every "+ "session), and what pass criterion is honest at num_generations = 2."),}@@ -442,15 +1009,43 @@ UNWIRED: dict[str, str] = {# actually use it. Splitting the group without packing the kernel, or packing# without splitting the group, each make things worse on their own.#-# Ordered LONGEST EXPECTED LEG FIRST, and that ordering is load-bearing.-# build_kernel.py hands this order to the driver as its start order and a card-# takes work greedily, so putting gptoss (384s, and the leg that sets the-# makespan) anywhere but first leaves the schedule unable to balance around it.-# Measured on run 32607621452: gptoss 384.1s, frontier 312.2s, canary 265.3s,-# control 262.2s, which this order packs as 384.1+262.2 = 646.3s against-# 312.2+265.3 = 577.5s. That is the optimal split of these four; the next best-# pairing is 649.4s, and perfect balance would be 611.9s, so the 68.8s of idle-# at the end is 34.4s of genuinely unavoidable imbalance and not a packing bug.+# gptoss sits THIRD, and that position is load-bearing. This is not the+# longest-first order it used to be, and it is not tidy either, so it needs its+# reasoning attached or the next reader will "fix" it back.+#+# The driver now prefetches the models on a lane that takes no card (see+# kaggle_prefetch.py). A prefetch only pays for the work it finishes BEFORE the+# leg that wants the model starts, so gptoss -- the only leg with a 12 GB+# download -- must not start at t=0. Third is first pick of the SECOND WAVE: it+# starts at ~190-220s, which is lead time the prefetch can spend, and a small+# leg is still running beside it.+#+# Simulated over every permutation, against the durations measured on the real+# one-kernel run 32611343797 (gptoss 384.1s, frontier 307.1s, canary 191.5s+# warm, control 179.0s warm) and a range of the unknown gpt-oss download time D:+#+# order prefetch worst (D 40..200) mean+# canary, control, gptoss, frontier yes 528.1s 504.1s+# control, canary, gptoss, frontier yes 540.6s 500.8s+# gptoss, frontier, canary, control no 563.1s 563.1s+# gptoss, frontier, canary, control yes 603.1s 603.1s+# frontier, canary, control, gptoss yes 651.1s 585.2s+#+# Three results, each contradicting something that sounds right:+#+# * Prefetching WITHOUT reordering is a REGRESSION. gptoss starts at t=0, so+# there is no window in front of it to fill, and it waits on a download it+# could have started itself.+# * gptoss LAST is worse than doing nothing for any D under ~120s. It is the+# longest leg, so ending on it leaves the other card idle for its whole+# ~284s. Standard LPT; "run the big model last" is the intuition to resist.+# * This order is never worse than the old one at any D tested, including+# D=40 where the prefetch barely helps. Chosen on WORST CASE rather than+# mean, deliberately, because D is not measured yet.+#+# If the prefetch fails outright the schedule degrades to ~568s against the old+# 563.1s: gptoss starts at ~184s and downloads for itself exactly as before.+# Costing 5s to maybe save 35-65s is why this is safe to ship before D is known.## control and canary stay in the same kernel, which is what their comparison# needs: same image, same driver, same hour. They no longer run on the two@@ -467,7 +1062,94 @@ UNWIRED: dict[str, str] = {# variable and one contrasting observation was not enough to blame a shared# host. It stays unwired rather than re-paired, since a leg passing one session# in three tells CI nothing either way.-KERNELS: tuple[tuple[str, ...], ...] = (("gptoss", "frontier", "canary", "control"),)+#+# `frontier` is REPLACED by `vision_fla_compile` rather than joined by it.+# frontier was "latest transformers and trl on Qwen2.5-0.5B"; the vision leg is+# latest transformers and trl on Qwen3.5-2B, and on top of that it asserts the+# vendored FLA kernels, the attention choice on Turing, a real vision training+# run with pixels on the card, the merged vision export, a Q8_0 GGUF with its+# mmproj sidecar and inference on the exported file. Everything frontier proved+# is a subset of that. Keeping both would spend a card on the smaller claim,+# and MAX_LEGS_PER_KERNEL is 5.+#+# Verified GREEN as a whole leg before wiring, on one T4:+# unsloth-probe-visleg-final-2b896b reports passed true with failures [],+# vision_failures [] and kernel_failures [], at 2.84GB peak reserved -- so it+# co-tenants comfortably rather than displacing anything.+KERNELS: tuple[tuple[str, ...], ...] = (+ ("canary", "control", "gptoss", "vision_fla_compile", "default"),+)+++# What the prefetch lane warms, in order, into the Kaggle image's default HF+# cache. See kaggle_prefetch.py for the mechanism and KERNELS above for why+# gptoss is third.+#+# The SMALL model first, which inverts what looks obvious. gpt-oss is bigger,+# but it is wanted by ONE leg whose own setup does not finish until ~160s, so a+# fetch that lands by then is early enough. Qwen2.5-0.5B gates THREE legs and+# the first of them starts at t=0, so it is the only fetch with no lead time in+# front of it at all. Fetching the big one first pushes the small one out past+# the moment the first leg is ready, delaying three legs to give one a head+# start it did not need.+#+# These must match what the legs LOAD, which is not the same thing as what+# they ASK FOR, and the difference cost a whole prefetch once already. The+# gptoss payload declares DEFAULT_MODEL = "unsloth/gpt-oss-20b", but that is an+# MXFP4 checkpoint and sm_75 cannot read MXFP4, so unsloth redirects to+# `-unsloth-bnb-4bit` at load time; run_gptoss_t4.py's own docstring records the+# redirect and the leg's output on run 32667451396 shows both strings. The+# first version of this list prefetched the declared name and therefore pulled+# 55.1 GB of a checkpoint no leg ever opens, while the leg downloaded the real+# one for itself exactly as before.+#+# Nothing at runtime notices this: a prefetch of the wrong repo downloads+# happily, warms a cache nobody reads, and reports success. The guard that was+# supposed to catch it compared against DEFAULT_MODEL and so agreed with the+# bug; it now knows about the redirect.+PREFETCH_REPOS: tuple[str, ...] = (+ # CRITICAL PATH FIRST, and that is a reversal of the original order.+ #+ # The earlier argument was gpt-oss first for MARGIN, because D -- how long+ # 12.5 GB takes -- was unknown and gptoss was the leg the schedule was built+ # around. D is now measured at 61.7s on kernel+ # unsloth-probe-prefetch-verify-9568-7a0bdd (~203 MB/s once the symlinked+ # blob double-count is removed), so every repo here lands inside ~90s+ # whatever the order, and the leg that needs one EARLIEST should not be the+ # one waiting.+ #+ # vision_fla_compile is that leg: it starts at t~21 and it sets the+ # makespan, so its 4.58 GB checkpoint is what a background lane should be+ # fetching while the legs are still installing.+ "unsloth/Qwen3.5-2B",+ # Default's model and the 4bit sibling `load_in_4bit=True` actually+ # resolves to. Both, because FLOAT_TO_INT_MAPPER redirects at load time and+ # warming only the name that appears in the args warms a cache the leg+ # never reads -- which downloads happily and saves nothing.+ "unsloth/Qwen3-0.6B",+ "unsloth/qwen3-0.6b-unsloth-bnb-4bit",+ # canary and control, which take the payload default.+ "unsloth/Qwen2.5-0.5B-Instruct",+ # Last now rather than first: gptoss is admitted only once a card empties,+ # around t~500 on the measured schedule, so it has the most slack of+ # anything here.+ "unsloth/gpt-oss-20b-unsloth-bnb-4bit",+)++# Declared name -> what actually gets loaded on an sm_75 card. Kept beside the+# list it corrects so the two cannot drift apart silently.+LOAD_REDIRECTS: dict[str, str] = {+ "unsloth/gpt-oss-20b": "unsloth/gpt-oss-20b-unsloth-bnb-4bit",+ # CASE MATTERS HERE, and it is not a typo. Two runs report+ # `resolved_checkpoint: unsloth/qwen3-0.6b-unsloth-bnb-4bit` in lower case+ # (all5-kernel-e28818 and defaultleg-s20-002d25), while gpt-oss above+ # resolves with its capitals intact. The HF cache keys on the literal+ # string -- `models--unsloth--qwen3-0.6b-unsloth-bnb-4bit` is a different+ # directory from `models--unsloth--Qwen3-0.6B-unsloth-bnb-4bit` -- so+ # prefetching the pretty spelling warms a directory the leg never reads and+ # the download happens twice. Copied from the report rather than typed.+ "unsloth/Qwen3-0.6B": "unsloth/qwen3-0.6b-unsloth-bnb-4bit",+}def expand_install(diff --git a/.github/scripts/kaggle_t4_ci/report.py b/.github/scripts/kaggle_t4_ci/report.pyindex 6d5491d0e14..d694deee783 100644--- a/.github/scripts/kaggle_t4_ci/report.py+++ b/.github/scripts/kaggle_t4_ci/report.py@@ -92,6 +92,42 @@ def version_table(reports: list) -> list[str]:return lines+# The label the Studio payload reports under. Duplicated in+# kaggle_studio_ci/report.py rather than shared: these two packages have no+# import relationship and both already ship a module called `report`, so a+# shared helper would mean putting one of them on the other's sys.path -- which+# is how `import report` starts resolving to the wrong file.+STUDIO_LABEL = "studio-gpu"+++def own_verdict(kernel_verdict: str, kernel_reason: str, reports: list, expect: int):+ """This reporter's verdict over ITS OWN payloads, not the kernel's.++ The launcher writes one verdict for the whole kernel, and since+ --with-studio that kernel holds two unrelated experiments. Reading the+ kernel verdict here means a failing training leg prints "Kaggle T4 smoke: FAIL"+ above a section listing zero failures, and a failing Studio payload prints+ the same over four green legs. Both are the misleading-red twin of the+ green tick that tested nothing, and both would send someone to read the+ wrong payload.++ So the verdict is recomputed from the filtered reports. The kernel reason+ is kept only when the two agree; otherwise it describes the other half.++ `infra` is deliberately not synthesised: with nothing of ours back, the+ kernel-level reason (quota, concurrency cap, a push that was throttled) is+ the only account of why, and it applies to every payload equally.+ """+ if not reports:+ return (kernel_verdict if kernel_verdict == "infra" else "partial"), kernel_reason+ failing = [r for r in reports if not r.get("passed")]+ if failing:+ return "fail", f"{len(failing)} of {len(reports)} payload(s) failed their assertions"+ if len(reports) < expect:+ return "partial", f"only {len(reports)} of {expect} payload(s) reported back"+ return "pass", f"all {len(reports)} payload(s) passed"++def render(report: dict) -> list[str]:lines = [f"#### payload `{report.get('label', '?')}` " f"model `{report.get('model', '?')}`",@@ -301,6 +337,66 @@ def kernel_log_text(evidence: Path) -> str:return "".join(chunks)+PREFETCH_SENTINEL = "KAGGLE_CI_PREFETCH"+++def prefetch_table(evidence: Path) -> list[str]:+ """What the prefetch lane actually achieved, from the kernel log.++ This is the instrument, not decoration. The gpt-oss download time was never+ measured -- an earlier estimate of "~282s" was subtraction, not measurement+ -- and the whole leg order is arranged around it. Putting the number in the+ job summary is what lets the next person confirm or reject the reorder+ without downloading an artifact, including the case where it did not pay+ for itself.++ Absent on a kernel built without the lane, which reads as no section at+ all rather than as a table of zeroes.+ """+ text = kernel_log_text(evidence)+ if not text:+ return []+ records = []+ for line in text.splitlines():+ marker = PREFETCH_SENTINEL + " "+ if marker in line:+ try:+ records.append(json.loads(line.split(marker, 1)[1]))+ except (ValueError, IndexError):+ continue+ if not records:+ return []+ lines = [+ "#### model prefetch",+ "",+ "| repo | ok | download s | MB/s | GB | transport | attempts |",+ "|---|---|---|---|---|---|---|",+ ]+ for r in records:+ gb = round((r.get("bytes") or 0) / 1e9, 2)+ lines.append(+ f"| `{r.get('repo', '?')}` | {'yes' if r.get('ok') else '**NO**'} | "+ f"{r.get('download_seconds') if r.get('download_seconds') is not None else '-'} | "+ f"{r.get('mb_per_s') if r.get('mb_per_s') is not None else '-'} | {gb} | "+ f"{r.get('transport', '?')} | {r.get('attempts', '?')} |"+ )+ failed = [r for r in records if not r.get("ok")]+ lines.append("")+ if failed:+ # Not a failure of the run. Said out loud anyway, because the schedule+ # assumes this lane worked: legs.KERNELS starts gptoss third to give it+ # a window, and if the window went unused the makespan is the ~568s+ # fallback rather than the ~500s the order was chosen for.+ lines.append(+ f"{len(failed)} of {len(records)} prefetch(es) failed. This does not fail "+ "the run -- the leg downloads the model itself -- but the leg order in "+ "`legs.KERNELS` is arranged around this lane working, so the makespan "+ "above is the fallback rather than the intended one."+ )+ lines.append("")+ return lines++def diagnostic_lines(evidence: Path, limit: int = 40) -> list[str]:"""The lines of the kernel log worth putting in front of a human.@@ -338,6 +434,15 @@ def main() -> int:reason = result.get("reason", "")reports = result.get("reports", [])+ # The Studio payload can share this kernel (see kaggle_t4_ci/build_kernel.py+ # --with-studio), and it emits its report through the same prefix, so it+ # arrives in this list. It is a different SHAPE -- assertions rather than a+ # per-step metric trace, no `config`, no `model` -- so rendering it here+ # produces a training leg made of question marks. kaggle_studio_ci/report.py+ # renders it properly; each reporter owns its own labels.+ reports = [r for r in reports if r.get("label") != STUDIO_LABEL]+ verdict, reason = own_verdict(verdict, reason, reports, args.expect)+header = {"pass": "### Kaggle T4 smoke: PASS","fail": "### Kaggle T4 smoke: FAIL",@@ -363,6 +468,7 @@ def main() -> int:)lines.append("")+ lines += prefetch_table(evidence)lines += version_table(reports)for report in reports:lines += render(report)diff --git a/.github/workflows/kaggle-t4-notebook-ci.yml b/.github/workflows/kaggle-t4-notebook-ci.ymlindex 7801847e191..de6bc3fe47e 100644--- a/.github/workflows/kaggle-t4-notebook-ci.yml+++ b/.github/workflows/kaggle-t4-notebook-ci.yml@@ -25,6 +25,23 @@# session. Longest first, because the longest leg is what sets the makespan and# a greedy schedule cannot balance around one it picks up last.#+# A FIFTH payload rides the same kernel and is not on that queue: the Studio GPU+# smoke, packed in by --with-studio and split into two halves. Its install --+# clone, pip, a Playwright browser -- takes no card at all and runs from t=0+# beside the training legs, so it finishes at ~400s inside a 646.3s card queue+# and costs nothing. Its GPU half then waits for the queue to DRAIN and takes+# both cards, because Studio loads a model and serves it rather than sharing a+# card with a training run. Measured makespan 763.3s against 646.3s without it.+#+# The install half is a gate as well as a lane: if it fails, the GPU half is+# skipped rather than run against a Studio that is not there, and the skip is+# reported rather than silent.+#+# This does NOT replace kaggle-t4-studio-gpu-ci.yml, whose paths filter and+# budget are untouched. A studio/** change still gets a dedicated run there;+# this is the same payload answering for the events THIS workflow already+# triggered on.+## This was two kernels of two, one leg per T4, and the reason it is one kernel# is NOT quota: a session bills its wall clock once rather than per card, so the# two shapes cost within a few percent of each other. It is that two kernels@@ -132,15 +149,31 @@# rest. Per-leg durations measured on run 32607621452: gptoss 384.1s, frontier# 312.2s, canary 265.3s, control 262.2s. Packed longest-first across two cards# that is 384.1 + 262.2 = 646.3s on one card against 312.2 + 265.3 = 577.5s on-# the other, so one kernel of four legs measures ~0.18 h:+# the other, so one kernel of four legs measures ~0.18 h.+#+# The Studio payload rides the same kernel (--with-studio below) and adds+# ~117s to that, not the ~0.75 h it costs standalone. Its install is CPU work+# -- clone, pip, a Playwright browser download -- so it runs on a lane of its+# own from t=0, finishes at ~400s against a 646.3s card queue, and is+# therefore FULLY hidden. What is left is its GPU half at ~200s, which starts+# once both cards are free. 646.3 + 117 = 763.3s, or ~0.21 h:# one kernel, four legs 0.18 h+# + the Studio payload, install hidden +0.03 h# TOTAL, expected ~0.25 h (rounded up)# The two-kernel shape this replaced measured 0.10 h + 0.13 h = 0.23 h, so# consolidating is roughly quota-neutral; the gain is the freed session, not-# the hours. The 0.25 h envelope is DELIBERATELY not lowered to match the new-# measurement. Every figure below is derived from it, the real cost moved-# DOWN so it remains an upper bound, and re-deriving a whole budget to book a-# 0.05 h saving would be a lot of arithmetic to make the reserve thinner.+# the hours. The 0.25 h envelope is unchanged and still an+# upper bound: 0.21 h measured against it. Every figure below is derived from+# that envelope, so leaving it alone is what keeps the sampling rate, the+# expected spend and the reserve below true without re-deriving all of them.+#+# Note what packing Studio in does NOT do: it does not widen the paths filter+# below. Studio's own thirteen paths stay on kaggle-t4-studio-gpu-ci.yml,+# which still runs standalone for a studio/** change and still owns the 35h+# share. The payload here is EXTRA coverage on events this workflow already+# answered -- an unsloth/** change now also says whether Studio still+# installs and serves on a T4 -- bought for 0.03 h because the install had+# nothing to contend with.## SAMPLING RATE, solved at the pessimistic end of the eligible range and# targeting 9h rather than the full 15, so a busy week does not spend the@@ -249,8 +282,23 @@ on:- '.github/scripts/kaggle_t4_ci/**'- '.github/workflows/kaggle-t4-notebook-ci.yml'- 'pyproject.toml'+ # NIGHTLY, for the legs that cannot be per-PR. The GRPO leg is the whole+ # reason this trigger exists: measured over nine T4 sessions it hits an+ # intermittent illegal memory access in vLLM's standby sleep on Turing in+ # FOUR of them. A 44% red in front of every PR, for a race no reader can act+ # on, is how a check gets switched off; at nightly cadence a clean run still+ # arrives most days and a red one costs nobody a merge.+ #+ # 03:17 rather than 03:00: every scheduled workflow on GitHub asks for the+ # hour mark, and this one has no reason to join that queue.+ schedule:+ - cron: '17 3 * * *'workflow_dispatch:inputs:+ legs:+ description: 'Comma-separated leg names to run instead of the wired KERNELS set. Empty means the wired set. The nightly uses this to run grpo, which is deliberately not per-PR'+ type: string+ default: ''force:description: 'Bypass the sampling gate (quota and concurrency still apply)'type: boolean@@ -267,6 +315,19 @@ on:description: 'unslothai/unsloth ref to install and test'type: stringdefault: ''+ slot:+ description: 'Which of the accounts two Kaggle session slots to occupy. Two dispatches naming different slots run side by side; anything else takes slot 1'+ type: choice+ options: ['1', '2']+ default: '1'+ shared_wheels:+ description: 'Build unsloth and unsloth_zoo once up front and install every leg from the wheel, instead of each leg resolving the git spec itself'+ type: boolean+ default: false+ studio_concurrent:+ description: 'Let the Studio GPU half share a card with a small leg instead of waiting for the queue to drain. ON by default: it takes about 1100s off the critical path, at the cost of Studio seeing one T4 rather than two. Set false to get the both-cards coverage back on a run that can afford the wall clock'+ type: boolean+ default: truepermissions:contents: read@@ -278,8 +339,18 @@ permissions:# result. Superseded runs that have not STARTED are still discarded (GitHub# keeps only the most recent pending run per group), which is where the saving# was anyway; only a run already spending is left alone.+#+# The slot belongs in THIS group too, not only in the job's. GitHub keeps at+# most one PENDING run per group and DISCARDS the older one when a newer run+# queues, so two dispatches on the same ref that were meant to hold Kaggle's+# two session slots at once instead cancel each other before either starts.+# That is not hypothetical: it killed run 32674255736, and a later A/B had to+# be run serially because a single push into this group would have discarded+# the pending arm. The job-level group carries the slot as well, and it is the+# one that actually serialises same-slot work; this one only has to stop+# distinct slots being collapsed into a single queue position.concurrency:- group: ${{ github.workflow }}-${{ github.ref }}+ group: ${{ github.workflow }}-${{ github.ref }}-${{ inputs.slot || '1' }}cancel-in-progress: falsejobs:@@ -385,7 +456,7 @@ jobs:run: |python .github/scripts/kaggle_t4_ci/gate.py \--percent 15 \- --force '${{ inputs.force || 'false' }}' \+ --force '${{ inputs.force || (github.event_name == 'schedule' && 'true' || 'false') }}' \--labels "$PR_LABELS" \--label-name kaggle-t4-ci \--event-action "$EVENT_ACTION" \@@ -473,7 +544,22 @@ jobs:# x one kernel is exactly Kaggle's 2-session cap, with no headroom, so the# capacity handling on the push path is load bearing rather than defensive.concurrency:- group: kaggle-t4-account-notebook+ # SLOTTED, because one group cannot express "at most two".+ #+ # A Kaggle account allows 2 concurrent GPU sessions and this job takes+ # one, so the cap has to be 2 and GitHub concurrency only offers 1 per+ # group. A single group therefore serialised everything -- and worse than+ # serialised: only ONE run may be PENDING in a group, so a second queued+ # run CANCELS the first rather than queueing behind it. That is what+ # killed run 32674255736, and it makes an A/B impossible to run at all.+ #+ # Two slots, chosen explicitly by the dispatcher, is the whole mechanism.+ # It is bounded by construction (the input offers 1 or 2 and nothing+ # else), so the account can never be asked for a third session, and two+ # dispatches that name different slots genuinely run side by side.+ # Everything that is not a dispatch has an empty input and lands in slot+ # 1, which is exactly the old behaviour for pushes and pull requests.+ group: kaggle-t4-account-notebook-${{ inputs.slot || '1' }}cancel-in-progress: falsesteps:# FIRST, ahead of even the audit hardening, and it is an `echo` so there@@ -563,7 +649,14 @@ jobs:- name: Test the harnessrun: |python -m pip install --quiet --index-url https://download.pytorch.org/whl/cpu torch- python -m pip install --quiet pytest transformers peft+ # Every third-party module tests/kaggle/test_*.py imports without a+ # pytest.importorskip guard. pillow and pyyaml arrived transitively+ # until now, which is luck rather than a dependency, and datasets did+ # not arrive at all: run 32925226213 ran 803 of 805 tests and reported+ # success on the step, because two vision-dataset guards raised+ # ModuleNotFoundError. test_harness_dependencies.py keeps this line and+ # that import set in agreement.+ python -m pip install --quiet pytest transformers peft datasets pillow pyyamlpython -c "import torch, sys; sys.exit(None if torch.version.cuda is None else 'torch %s is built for CUDA %s: the CPU-index install was skipped as already satisfied' % (torch.__version__, torch.version.cuda))"python -m pytest tests/kaggle -q@@ -735,6 +828,29 @@ jobs:id: buildif: steps.ref.outputs.stand_down != 'true' && steps.pins.outputs.stand_down != 'true' && steps.stepcount.outputs.stand_down != 'true'env:+ # On the nightly this is 'grpo,multi_gpu'; on every other trigger it+ # is empty and the wired set runs. `github.event.inputs` is null on a+ # schedule, so the default has to come from the fallback rather than+ # from the input's own default.+ #+ # multi_gpu joined grpo here after the ab3 A/B: it PASSES, but it+ # costs makespan (+172.4s and +39.7s on two same-commit pairs, B+ # slower 2 of 2), and the brief for it was multi-GPU coverage at no+ # wall-clock cost. The two legs also serialise cleanly rather than+ # contending: grpo declares 13.8GB and admits no co-tenant, and+ # multi_gpu reserves on every card, so neither ever starts beside the+ # other.+ #+ # latest_compile joined them once unsloth-zoo #1103 merged, and NOT+ # the per-PR kernel its UNWIRED note promised. The note predates the+ # only number that decides it: the leg's own DONE record is 1323.0s+ # (unsloth-probe-lcleg-tmpdir-ac53ca) and at 12.73GB peak it admits+ # no co-tenant, so it needs a card to itself for 22 minutes. The+ # per-PR kernel's only slack is gpu1's 776.3s idle block while Studio+ # holds gpu0, which is not half enough, so wiring it there would add+ # roughly 580s to a 2101.8s makespan. It serialises here for the same+ # reason it does not fit there.+ LEG_LIST: ${{ inputs.legs || (github.event_name == 'schedule' && 'grpo,multi_gpu,latest_compile' || '') }}SKIP_BAND: ${{ inputs.skip_reference_band }}# The VALIDATED, normalised count from the step above, not the raw# dispatch string. check_steps.parse_steps accepts "+10", "010" and@@ -760,14 +876,57 @@ jobs:SKIP='--skip-reference'echo "::warning title=Reference band check skipped::this run is $MAX_STEPS steps and the committed reference was captured at $REF_STEPS, which are not comparable. The band check is off for this run; everything else the control leg asserts still applies."fi+ # Empty unless asked for. build_kernel defaults AFTER_GPU_CONCURRENT+ # to False, so an unset input and a missing flag agree -- but they+ # agreed silently once before, on run 32674263571, where the flag+ # existed in the CLI and nothing here passed it: the "variant" ran the+ # control's schedule and the A/B compared a configuration against+ # itself. test_the_workflow_can_actually_reach_studio_concurrent is+ # what stops that recurring.+ # Off unless asked for. Measured on run 32689629906 the wheels helped+ # the leg that runs ALONE (gpt-oss install 191.2s -> 152.6s) and cost+ # the three that run CONCURRENTLY (149-163s -> 319-334s). That run+ # changed the caches at the same time, so neither effect is+ # attributable yet, which is why this is a flag rather than a+ # decision.+ SHARED_WHEELS=''+ if [ "${{ inputs.shared_wheels }}" = "true" ]; then+ SHARED_WHEELS='--shared-wheels'+ fi+ # Default ON, and `inputs` is null on push and on a schedule, so the+ # fallback rather than the input default is what decides those. The+ # measurement behind it: Studio's GPU half is ~1100s and it used to+ # start only once every leg had freed both cards, on top of a+ # ~1200s leg phase. Sharing a card moves it alongside instead.+ STUDIO_CONCURRENT='--studio-concurrent'+ if [ "${{ inputs.studio_concurrent }}" = "false" ]; then+ STUDIO_CONCURRENT=''+ fi+ if [ -n "$STUDIO_CONCURRENT" ]; then+ echo "::notice title=Studio shares a card::the Studio GPU half is pinned to one T4 beside a small leg, so this run does NOT cover Studio's own choice between two cards. The report records tensor_split_over_two_cards=false rather than claiming it."+ fi+ # A leg list REPLACES --all-kernels rather than filtering it: the+ # kernel plan and the payload count both come out of the same call,+ # so a filter applied afterwards would leave the launcher expecting+ # payloads that were never built.+ if [ -n "$LEG_LIST" ]; then+ KERNEL_SELECT="--legs $LEG_LIST"+ echo "::notice title=Leg override::running $LEG_LIST instead of the wired set"+ else+ KERNEL_SELECT="--all-kernels"+ fipython .github/scripts/kaggle_t4_ci/build_kernel.py \--payload-dir tests/kaggle/t4_smoke \--out kernel \- --all-kernels \+ $KERNEL_SELECT \--unsloth-ref '${{ steps.ref.outputs.ref }}' \--zoo-ref '${{ steps.pins.outputs.zoo_ref }}' \--smoke-args "--max-steps $MAX_STEPS" \- --per-run-timeout 3300 \+ --with-studio \+ --studio-args "--max-steps 8" \+ --per-run-timeout 3900 \+ $STUDIO_CONCURRENT \+ $SHARED_WHEELS \$SKIP# kernel-timeout-sec is passed to Kaggle at push time as a BACKSTOP, not@@ -857,6 +1016,21 @@ jobs:--poll-every 60 \--deadline-epoch "$(( JOB_START_EPOCH + JOB_TIMEOUT_MINUTES * 60 ))"+ # The launcher collects executed notebooks and the kernel log and nothing+ # else, deliberately. The Studio payload therefore encodes its+ # screenshots and logs into its own cell output; this puts them back.+ # Lifted from kaggle-t4-studio-gpu-ci.yml along with the payload itself.+ #+ # It is safe on a run that carried no Studio payload: the collector finds+ # no encoded evidence and writes nothing.+ - name: Unpack the Playwright evidence+ if: always() && steps.recheck.outputs.should_run == 'true'+ continue-on-error: true+ run: |+ python .github/scripts/kaggle_studio_ci/collect_evidence.py \+ --evidence kaggle_evidence \+ --outdir studio_evidence+# always(): the evidence is most valuable when something went wrong, and# the executed notebooks are kilobytes.#@@ -881,6 +1055,7 @@ jobs:name: kaggle-t4-evidencepath: |kaggle_evidence/**+ studio_evidence/**kernel*.ipynb# 14 rather than the repo's usual 7: this job runs on a minority of# commits, so two consecutive samples can be a fortnight apart, and@@ -905,4 +1080,18 @@ jobs:run: |python .github/scripts/kaggle_t4_ci/report.py \--evidence kaggle_evidence \- --expect '${{ steps.build.outputs.payloads || 4 }}'+ --expect '${{ steps.build.outputs.legs || 4 }}'++ # Two reporters over ONE evidence directory, each rendering the labels it+ # owns: the four training legs above, the Studio assertions here. They+ # are not merged because the two payload shapes are not comparable -- a+ # per-step metric trace against a list of assertions -- and a single+ # renderer over both produces a training leg made of question marks.+ #+ # Its exit code is the same contract as the T4 reporter's: red only for a+ # payload that ran and failed its assertions.+ - name: Report Studio+ if: always() && steps.recheck.outputs.should_run == 'true'+ run: |+ python .github/scripts/kaggle_studio_ci/report.py \+ --evidence kaggle_evidencediff --git a/.github/workflows/kaggle-t4-studio-gpu-ci.yml b/.github/workflows/kaggle-t4-studio-gpu-ci.ymlindex 04f7f3ef6d4..d9628e7c08b 100644--- a/.github/workflows/kaggle-t4-studio-gpu-ci.yml+++ b/.github/workflows/kaggle-t4-studio-gpu-ci.yml@@ -377,8 +377,13 @@ jobs:- name: Build the kernel notebookenv:MAX_STEPS: ${{ inputs.max_steps || '8' }}- CHAT_MODEL: ${{ inputs.chat_model || 'unsloth/Qwen3.5-2B-GGUF' }}- TRAIN_MODEL: ${{ inputs.train_model || 'unsloth/Qwen2.5-0.5B-Instruct' }}+ # MTP-GGUF, not the plain GGUF: multi-token prediction is a distinct+ # serving path in llama.cpp and the plain repo does not exercise it.+ CHAT_MODEL: ${{ inputs.chat_model || 'unsloth/Qwen3.5-2B-MTP-GGUF' }}+ # Qwen3.5-2B rather than Qwen2.5-0.5B-Instruct, so Studio trains the+ # model this CI actually cares about. Confirmed loadable on a T4 in+ # 4bit by unsloth-probe-vision-recon-c76ea3.+ TRAIN_MODEL: ${{ inputs.train_model || 'unsloth/Qwen3.5-2B' }}SKIP_UI: ${{ inputs.skip_ui }}run: |ARGS="--max-steps $MAX_STEPS --chat-model $CHAT_MODEL --train-model $TRAIN_MODEL"diff --git a/tests/kaggle/studio_gpu/run_studio_gpu.py b/tests/kaggle/studio_gpu/run_studio_gpu.pyindex cc6232d61c3..84686e099aa 100644--- a/tests/kaggle/studio_gpu/run_studio_gpu.py+++ b/tests/kaggle/studio_gpu/run_studio_gpu.py@@ -66,6 +66,8 @@ import base64import ioimport jsonimport os+import re+import secrets as secrets_moduleimport shutilimport subprocessimport sys@@ -193,6 +195,65 @@ def nvidia_used_mib() -> float | None:return total if seen else None+def cli_run_gpu_failure(+ apps_before: dict[int, int] | None,+ apps_after: dict[int, int] | None,+ baseline: float | None,+ settled: float | None,+) -> tuple[str | None, dict]:+ """Did a model actually reach the card? Returns (failure or None, detail).++ WHICH RULER, and this one has now been wrong in both directions.++ The device total is a SHARED reading. On kernel+ unsloth-probe-studio-full2-815a0c it was sampled too early and read 0.0 MiB+ on a server that did have the weights; the fix was to sample after a served+ completion. On unsloth-probe-full-concurrent-417238 it read **-182.0 MiB**+ -- and the same report carried `compute_apps {"6841": 2628}`, so the model+ was on the card and 2.6 GB of it. That run was the first with+ --studio-concurrent, so a training leg shared the card and freed memory+ inside the window. A shared counter cannot attribute, and subtracting two of+ its samples is not a measurement of THIS process.++ So the verdict comes off the pids that APPEARED during the window, which is+ per-process and immune to a co-tenant. A pid already on the card before the+ launch is excluded, or a co-tenant holding gigabytes would satisfy the claim+ on its own -- which is the same failure in a new costume.++ The device delta is still recorded, and is still the fallback for an+ nvidia-smi that answers a total but cannot enumerate processes.+ """+ detail: dict = {}+ if baseline is not None and settled is not None:+ detail["vram_delta_mib"] = round(settled - baseline, 1)++ if apps_after is None:+ if baseline is None or settled is None:+ return "nvidia-smi did not answer, so GPU use is unmeasured", detail+ if settled - baseline < 200.0:+ return (+ f"device VRAM grew by {settled - baseline:.1f} MiB across the "+ f"launch and a served completion, and nvidia-smi could not "+ f"enumerate processes to attribute it -- `unsloth run` served "+ f"from the CPU"+ ), detail+ return None, detail++ before = apps_before or {}+ appeared = {pid: mib for pid, mib in apps_after.items() if pid not in before}+ detail["compute_apps_appeared"] = appeared+ grew = sum(appeared.values())+ detail["process_vram_mib"] = grew+ if grew < 200.0:+ return (+ f"no process appeared on the GPU holding more than {grew} MiB "+ f"across the launch and a served completion (before "+ f"{sorted(before)}, after {sorted(apps_after)}) -- `unsloth run` "+ f"served from the CPU"+ ), detail+ return None, detail++def nvidia_compute_apps() -> dict[int, int] | None:proc = run(["nvidia-smi", "--query-compute-apps=pid,used_gpu_memory", "--format=csv,noheader,nounits"],@@ -203,14 +264,60 @@ def nvidia_compute_apps() -> dict[int, int] | None:return parse_compute_apps(proc.stdout)+def visible_device_indices() -> list[int] | None:+ """The physical card indices CUDA_VISIBLE_DEVICES exposes, or None if unset.++ An empty string is a deliberate "no cards", which is different from unset+ and must not read as "all of them".+ """+ raw = os.environ.get("CUDA_VISIBLE_DEVICES")+ if raw is None:+ return None+ out: list[int] = []+ for part in raw.split(","):+ part = part.strip()+ if not part:+ continue+ try:+ out.append(int(part))+ except ValueError:+ # A UUID form (GPU-xxxx) selects a card this cannot map to an+ # nvidia-smi row index. Returning None would report every card as+ # usable, which is the failure being fixed, so an unparseable entry+ # counts as one card rather than as all of them.+ out.append(-1)+ return out++def gpu_inventory() -> list[str]:+ """The cards THIS PROCESS can use, not the cards the box has.++ `nvidia-smi` enumerates PHYSICAL devices and ignores CUDA_VISIBLE_DEVICES,+ and reading it as "what is available" produced a false claim on kernel+ unsloth-probe-full-concurrent-417238. build_kernel.py pins every payload+ with `CUDA_VISIBLE_DEVICES = str(gpu_index)`, and under --studio-concurrent+ that includes Studio -- so the run recorded `cards_visible: 2` and+ `tensor_split_over_two_cards: True` for a server that had ONE card, and sent+ `tensor_split: [1.0, 1.0]` asking llama.cpp to split across a device that+ was not there. It loaded anyway, so the assertion passed green.++ That field exists precisely to stop a check keeping its name while testing+ less. It was sized from the wrong instrument and did exactly that.+ """proc = run(["nvidia-smi", "--query-gpu=name,memory.total,compute_cap", "--format=csv,noheader"],timeout = 60,)if proc.returncode != 0:return []- return [line.strip() for line in proc.stdout.splitlines() if line.strip()]+ rows = [line.strip() for line in proc.stdout.splitlines() if line.strip()]+ visible = visible_device_indices()+ if visible is None:+ return rows+ # Index into the physical rows where we can; an index nvidia-smi does not+ # have, or the UUID sentinel, still counts as one card so the COUNT stays+ # right even when the description cannot be recovered.+ return [rows[i] if 0 <= i < len(rows) else "visible GPU (details unavailable)" for i in visible]def log_paths(server_log: Path, studio_home: Path) -> list[Path]:@@ -322,10 +429,44 @@ class Payload:# artifact", so every log that leaves this machine is scrubbed of it.self.secrets: set[str] = set()+ # Resolved and registered HERE, before anything can log it. `auto`+ # mints a fresh one per run rather than carrying a constant, because a+ # constant in a repo is a credential whether or not it is ever reachable.+ if self.args.studio_password == "auto":+ self.args.studio_password = "ci-" + secrets_module.token_urlsafe(18)+ if self.args.studio_password:+ self.secrets.add(self.args.studio_password)+# ---------------------------------------------------------------- reportdef record(self, name: str, passed: bool, detail: dict) -> bool:- entry = {"name": name, "passed": bool(passed), **detail}+ # Per-assertion wall clock, because this payload is now the longest+ # thing in the kernel and nothing said where the time went. Measured on+ # unsloth-probe-full-concurrent-417238: 1487.5s across 19 assertions,+ # and the report carried no breakdown at all, so every question about+ # shortening Studio was guesswork.+ #+ # It is elapsed-since-the-previous-record, NOT a timer around the+ # assertion body, and the name says so. The assertions run back to+ # back, so the two are the same to within the bookkeeping between them;+ # the first entry measures from process start, which is the setup+ # before any assertion and is worth seeing rather than hiding.+ #+ # Both reads are `getattr` with a default: `record` is driven directly+ # by several CPU guards against stub objects that have no clock, and a+ # hard `self.started` turns every one of them into an AttributeError+ # about timing instead of a result about the rule they test.+ now = time.time()+ started = getattr(self, "started", now)+ previous = getattr(self, "_last_record_at", started)+ self._last_record_at = now+ entry = {+ "name": name,+ "passed": bool(passed),+ "seconds_since_previous": round(now - previous, 1),+ "at_seconds": round(now - started, 1),+ **detail,+ }self.assertions.append(entry)for reason in detail.get("failures", []) or []:self.failures.append(f"{name}: {reason}")@@ -431,7 +572,24 @@ class Payload:breakelse:head = [sys.executable, "-c", "from unsloth_cli import app; app()"]- return head + ["studio", "-H", "127.0.0.1", "-p", str(self.args.port)]+ cmd = head + ["studio", "-H", "127.0.0.1", "-p", str(self.args.port)]+ if self.args.studio_password:+ # The HEADLESS path, and it is a feature rather than a convenience+ # here: `--password` sets the INITIAL admin password when none is+ # set yet, which is exactly the shape a server started by a script+ # is in. Without it the only way in is the bootstrap password+ # Studio seeds into a file and prints to its own log, and a+ # deployment that has to read a log to log in is not one anybody+ # scripts twice.+ #+ # The value is generated per run and registered as a secret before+ # this is ever called, so it is scrubbed out of every log and+ # evidence bundle that leaves the machine. It is still visible in+ # this session's process list, which the flag's own help says; that+ # is acceptable for a single-tenant CI kernel and would not be on a+ # shared host.+ cmd += ["--password", self.args.studio_password]+ return cmddef start_server(self) -> bool:# An absent auth directory is what re-seeds the bootstrap password;@@ -504,8 +662,25 @@ class Payload:return self.scrub(" | ".join(text.splitlines()[-lines:]))def authenticate(self) -> bool:- path = self.studio_home / "auth" / ".bootstrap_password"failures: list[str] = []+ if self.args.studio_password:+ # Log in with the password we PASSED, which is the assertion: if+ # --password had been ignored, Studio would have seeded a bootstrap+ # password instead and this login would fail. A run that fell back+ # to the bootstrap on failure would pass while proving the flag does+ # nothing, so there is deliberately no fallback.+ try:+ self.studio.login(self.args.studio_password)+ except StudioError as exc:+ failures.append(+ f"--password was passed to `unsloth studio` and logging in "+ f"with it failed, so the flag did not take effect: {exc}"+ )+ return self.record(+ "authenticate", not failures, {"source": "--password", "failures": failures}+ )++ path = self.studio_home / "auth" / ".bootstrap_password"if not path.is_file():failures.append(f"no bootstrap password was seeded at {path}")return self.record("authenticate", False, {"failures": failures})@@ -517,7 +692,9 @@ class Payload:self.studio.login(password)except StudioError as exc:failures.append(str(exc))- return self.record("authenticate", not failures, {"failures": failures})+ return self.record(+ "authenticate", not failures, {"source": "bootstrap", "failures": failures}+ )def server_alive(self) -> bool:return self.proc is not None and self.proc.poll() is None@@ -646,6 +823,19 @@ class Payload:f"(a clamp to the model's layer count looks like this)")+ # Speculative decoding, read off the status rather than inferred from+ # the repo name. Pointing --chat-model at an MTP repo does NOT prove the+ # drafter engaged: if the companion is missing, llama.cpp was built+ # without MTP, or the drafter was downgraded for VRAM, the main GGUF+ # loads and generates exactly as it does otherwise and every assertion+ # here stays green. `spec_fallback_reason` is what names which of those+ # happened, so it is recorded whether or not it fires.+ spec = {+ "drafter_kind": status_body.get("spec_drafter_kind"),+ "fallback_reason": status_body.get("spec_fallback_reason"),+ "supports_mtp": status_body.get("llama_cpp_supports_mtp"),+ }+return {"model": model_path,"variant": variant,@@ -653,6 +843,7 @@ class Payload:"effective_gpu_layers": effective,"llama_server_pids": server_pids,"device_vram_delta_mib": None if delta is None else round(delta, 1),+ "spec_decoding": spec,"evidence": verdict["evidence"],"positives": verdict["positives"],"failures": failures,@@ -688,8 +879,368 @@ class Payload:detail["failures"].append(f"/v1/chat/completions returned HTTP {code}")elif not text.strip():detail["failures"].append("the model on the GPU returned empty content")++ # The MTP claim, checked rather than assumed from the repo name.+ # An MTP repo whose drafter never engaged serves ordinary decoding+ # and passes everything above it, so selecting the repo is a request+ # and this is the result. The reason is named rather than reported+ # as "MTP off", because "llama.cpp has no MTP support" and "the+ # drafter was downgraded for VRAM" are different findings and only+ # one of them is about this leg.+ spec = detail.get("spec_decoding") or {}+ if "MTP" in (self.args.chat_model or "").upper() and spec.get("drafter_kind") != "mtp":+ detail["failures"].append(+ f"--chat-model is {self.args.chat_model}, which was chosen to "+ f"exercise multi-token prediction, and Unsloth reports "+ f"spec_drafter_kind={spec.get('drafter_kind')!r} "+ f"(fallback_reason={spec.get('fallback_reason')!r}, "+ f"llama_cpp_supports_mtp={spec.get('supports_mtp')!r}). The "+ f"model served ordinary decoding, so nothing here covers the "+ f"MTP path and pointing at the MTP repo bought nothing"+ )return self.record("gpu_inference", not detail["failures"], detail)+ def assert_server_flags(self) -> bool:+ """Reload with a quantized KV cache and a two-card split, and check what+ was APPLIED rather than what was asked for.++ Studio's own status distinguishes the two, which is the point: it+ carries `cache_type_kv` for the live server alongside a coverage field+ and a reason for anything it could not apply. So a request that was+ silently dropped is visible, and a check that only asserted "the load+ succeeded" would pass on exactly that.++ `tensor_split` over two T4s is the flag the brief asks about and the+ one nothing here has ever exercised. The split is sized to the cards+ that are VISIBLE, because --studio-concurrent pins this half to one+ card so it can share with a training leg, and the report says which of+ the two it did: a single-card run records+ `tensor_split_over_two_cards: false` with a note rather than passing+ under the same name. A check that keeps its name while quietly testing+ less is the failure this file exists against.++ The context length is pinned to `--studio-ctx` (2048 by default) rather+ than left at the model default, because an unconstrained context on a+ 14.56GB card is how a KV-cache test turns into an OOM about something+ else.+ """+ failures: list[str] = []+ cards = gpu_inventory()+ detail: dict = {"requested": {}, "cards_visible": len(cards)}+ body = {+ "model_path": self.args.chat_model,+ "is_lora": False,+ "max_seq_length": self.args.studio_ctx,+ "gpu_memory_mode": "manual",+ "gpu_layers": self.args.gpu_layers,+ "force": True,+ # q8_0 rather than a fancier width: it is the one every llama.cpp+ # build supports, so a refusal here is about the MODEL's cache+ # layout and not about the binary.+ "cache_type_kv": "q8_0",+ # One weight per VISIBLE card, even. The values are relative+ # weights, not byte counts.+ #+ # Sized rather than hardcoded to two, because --studio-concurrent+ # pins this half to a single card so it can share with a training+ # leg: build_kernel.py's run_one sets CUDA_VISIBLE_DEVICES to the+ # card it was admitted on. Sending [1.0, 1.0] to a one-card server+ # asks llama.cpp to split across a device that is not there, and+ # what comes back is a failure about the load rather than about the+ # flag.+ "tensor_split": [1.0] * max(1, len(cards)),+ }+ if self.args.chat_variant:+ body["gguf_variant"] = self.args.chat_variant+ detail["requested"] = {+ k: body[k] for k in ("max_seq_length", "cache_type_kv", "tensor_split")+ }+ # STATED, not silent. Under --studio-concurrent this half runs on one+ # card so it can share with a training leg, and a split over one device+ # is not the two-card flag the brief asks about. Recording it as+ # exercised when it was not is how a check keeps its name and loses its+ # meaning; a reader of this report can see which machine it ran on.+ detail["tensor_split_over_two_cards"] = len(cards) >= 2+ if len(cards) < 2:+ detail["tensor_split_note"] = (+ f"only {len(cards)} card visible, so the two-card tensor_split "+ f"was NOT exercised; the KV-cache type, the context pin and the "+ f"GPU residency below still were"+ )++ try:+ self.studio.expect("POST", "/api/inference/load", body, timeout = self.args.load_timeout)+ except StudioError as exc:+ failures.append(f"loading with server flags failed: {exc}"[:600])+ detail["failures"] = failures+ return self.record("server_flags", False, detail)++ code, status_body = self.studio.get("/api/inference/status")+ detail["status_code"] = code+ status = status_body if isinstance(status_body, dict) else {}+ applied = {+ k: status.get(k)+ for k in (+ "cache_type_kv",+ "context_length",+ "max_context_length",+ "gpu_layers",+ "is_mlx",+ )+ }+ detail["applied"] = applied++ # 1. the KV cache type actually in force. Reported rather than asserted+ # equal: a model whose cache layout cannot be quantized is entitled+ # to refuse, and Studio says so. What is NOT acceptable is silence.+ got_cache = (applied.get("cache_type_kv") or "").lower()+ if not got_cache:+ failures.append(+ "the status reports no cache_type_kv at all after a load that "+ "asked for q8_0, so there is no way to tell whether the request "+ "was honoured, downgraded or ignored"+ )+ elif got_cache != "q8_0":+ # Not a failure by itself -- but it must come with a reason.+ reason = status.get("kv_quant_reason") or status.get("mlx_kv_quant_reason")+ detail["cache_downgrade_reason"] = reason+ if not reason:+ failures.append(+ f"q8_0 was requested and {got_cache!r} is in force, with no "+ f"reason given. A silent downgrade is the failure this check "+ f"exists for"+ )++ # 2. the context length is the one that was pinned. llama-server admits+ # a prompt on n_ctx alone, so a server running at the model default+ # behaves differently from one at 2048 and the difference is+ # invisible in a chat response.+ ctx = applied.get("context_length")+ if ctx is None:+ failures.append("the status reports no context_length")+ elif int(ctx) > self.args.studio_ctx:+ failures.append(+ f"context_length is {ctx}, above the {self.args.studio_ctx} that "+ f"was requested, so the pin did not take"+ )++ # 3. it is still on the GPU. A tensor split that fell back to CPU would+ # otherwise report a healthy server and prove nothing about either+ # card.+ used = nvidia_used_mib()+ detail["gpu_used_mib"] = used+ if used is not None and used < 200:+ failures.append(+ f"only {used} MiB of GPU memory is in use after a "+ f"{len(cards)}-card tensor_split load, so this is running on "+ f"the CPU"+ )++ detail["failures"] = failures+ return self.record("server_flags", not failures, detail)++ def assert_compaction(self) -> bool:+ """A conversation past the window must COMPACT, and a short one must not.++ The context length is pinned to `--studio-ctx` by `assert_server_flags`+ immediately above, so this is the one place the payload knows what the+ window is and can overflow it deliberately.++ Studio reports the fit on the completion itself, as `context_truncated`+ with `dropped_messages`, so the claim is readable without a browser.+ The rule is a PAIR, and the second half is what makes the first mean+ anything:++ * a conversation built past the budget comes back 200 with+ `dropped_messages > 0` -- it was shortened, not refused;+ * a two-message conversation comes back with nothing dropped.++ Asserting only the first passes on a server that reports truncation+ unconditionally, which is indistinguishable from working and is the+ failure this file keeps being caught by. Asserting only the second+ passes on a server that never compacts at all and returns a+ context-length error instead.+ """+ failures: list[str] = []+ detail: dict = {}++ def _dropped(body) -> int | None:+ if not isinstance(body, dict):+ return None+ truncation = body.get("context_truncated")+ if not isinstance(truncation, dict):+ return 0+ try:+ return int(truncation.get("dropped_messages") or 0)+ except (TypeError, ValueError):+ return 0++ # Distinct filler, so nothing can be deduplicated or cached into+ # fitting. Sized past `studio_ctx` by a wide margin rather than a+ # narrow one: the budget subtracts the reply reserve and the template's+ # own framing, so a prompt that merely equals the context length is not+ # reliably over it.+ long_messages = []+ for i in range(40):+ long_messages.append(+ {+ "role": "user",+ "content": f"Fact {i}: the {i}th token of this transcript is "+ + " ".join(f"w{i}x{j}" for j in range(40)),+ }+ )+ long_messages.append({"role": "assistant", "content": f"Noted fact {i}."})+ long_messages.append({"role": "user", "content": "Reply with one word."})++ # `context_overflow` defaults to "error", and that default is CORRECT:+ # a conversation past the window comes back 400 with+ # code=context_length_exceeded so a client's own trim loop can see it.+ # Compaction is a policy you ask for, and asking for it is the thing+ # under test. Measured the hard way on kernel+ # unsloth-probe-studio-full2-815a0c, where this assertion failed a+ # documented default. "truncate_oldest" is the policy that applies to a+ # plain chat; "truncate_middle" is limited to client-tool and+ # response_format passthrough (studio/backend/models/inference.py).+ code, body = self.chat(long_messages, max_tokens = 32, context_overflow = "truncate_oldest")+ detail["context_overflow"] = "truncate_oldest"+ detail["long_status"] = code+ detail["long_messages"] = len(long_messages)+ detail["long_truncation"] = (+ (body or {}).get("context_truncated") if isinstance(body, dict) else None+ )+ long_dropped = _dropped(body)+ if code != 200:+ failures.append(+ f"an over-length conversation returned HTTP {code} instead of "+ f"being compacted: {str(body)[:300]}"+ )+ elif not long_dropped:+ failures.append(+ f"a {len(long_messages)}-message conversation against a "+ f"{self.args.studio_ctx}-token window dropped nothing, so no "+ f"compaction happened and the reply used a prompt nobody sized"+ )++ # The POLICY control: the same over-length conversation with the+ # default `context_overflow` must be REFUSED. Without this, the check+ # above passes on a server that compacts everything regardless of what+ # was asked for, and the field name in the request would be decorative.+ code, body = self.chat(long_messages, max_tokens = 32)+ detail["long_status_default_policy"] = code+ # The CODE, not just the 400. `openai_error_body` always emits the key+ # and leaves it None for an unclassified failure, so a validation error+ # or a code-less generation error is also a 400 and would satisfy a+ # status-only check while proving nothing about the overflow semantics+ # this failure text promises.+ refusal_code = None+ if isinstance(body, dict):+ refusal_code = (body.get("error") or {}).get("code")+ detail["long_default_policy_code"] = refusal_code+ if code != 400 or refusal_code != "context_length_exceeded":+ failures.append(+ f"with the default context_overflow the same over-length "+ f"conversation returned HTTP {code} with error code "+ f"{refusal_code!r}, not 400 with code=context_length_exceeded. "+ f"That code is what a client's own trim loop detects, and if "+ f"everything is compacted anyway then asking for "+ f"truncate_oldest above proved nothing"+ )++ # The LENGTH control, and it is not optional either: without it a+ # server that always claims truncation passes the first check.+ code, body = self.chat([{"role": "user", "content": "Say hi."}], max_tokens = 16)+ detail["short_status"] = code+ short_dropped = _dropped(body)+ detail["short_dropped"] = short_dropped+ if code != 200:+ failures.append(f"the short control returned HTTP {code}")+ elif short_dropped:+ failures.append(+ f"a two-message conversation reported {short_dropped} dropped "+ f"messages, so the field is not responsive to length and the "+ f"check above proves nothing"+ )++ detail["failures"] = failures+ return self.record("compaction", not failures, detail)++ def assert_api_key(self) -> bool:+ """Mint an API key and DRIVE it, which is the half that can be wrong.++ Creating a key proves the endpoint returns a string. Whether that+ string authenticates anything is a separate question, and the answer+ that matters: an API key Studio issues and then rejects is worse than+ no API key, because the failure surfaces in a user's integration rather+ than here.++ Three claims, in order of what each rules out:++ 1. the key is minted and returned ONCE (the raw value is not+ retrievable later, by design);+ 2. a request carrying ONLY that key succeeds -- the session bearer+ token is set aside for the call, or this would pass on the token and+ say nothing about the key;+ 3. a request carrying a corrupted key FAILS. Without that, a server+ that ignores the header entirely passes claim 2.+ """+ failures: list[str] = []+ detail: dict = {}+ raw_key = None+ try:+ code, body = self.studio.post(+ "/api/auth/api-keys", {"name": f"kaggle-ci-{int(time.time())}"}+ )+ detail["create_status"] = code+ if code >= 400 or not isinstance(body, dict):+ failures.append(f"could not create an API key: {code} {str(body)[:200]}")+ else:+ raw_key = body.get("key")+ detail["has_key"] = bool(raw_key)+ detail["key_metadata"] = {+ k: v+ for k, v in (body.get("api_key") or {}).items()+ if k in ("id", "name", "is_active", "expires_at")+ }+ if not raw_key:+ failures.append("the create response carried no raw key")+ except BaseException as exc: # noqa: BLE001+ failures.append(f"creating an API key raised: {type(exc).__name__}: {exc}"[:300])++ if raw_key:+ # Registered before it is used anywhere, so it cannot reach a log.+ self.secrets.add(raw_key)+ saved = self.studio.token+ try:+ # The key AS the bearer, with the session token set aside. If+ # the session token were left in place this would pass on the+ # token and prove nothing.+ self.studio.token = raw_key+ code, body = self.studio.get("/api/auth/api-keys")+ detail["key_auth_status"] = code+ if code >= 400:+ failures.append(+ f"the API key Studio just issued does not authenticate: "+ f"{code} {str(body)[:200]}"+ )++ # And a corrupted key must be REJECTED. Without this, a server+ # that ignores the header entirely passes the check above.+ self.studio.token = raw_key[:-4] + "0000" if len(raw_key) > 8 else "bogus"+ code, _ = self.studio.get("/api/auth/api-keys")+ detail["bad_key_status"] = code+ if code < 400:+ failures.append(+ f"a corrupted API key was accepted ({code}), so the "+ f"check above passes whatever is sent"+ )+ except BaseException as exc: # noqa: BLE001+ failures.append(f"driving the API key raised: {type(exc).__name__}: {exc}"[:300])+ finally:+ self.studio.token = saved++ detail["failures"] = failures+ return self.record("api_key", not failures, detail)+def assert_tool_calling(self) -> bool:failures: list[str] = []detail: dict = {}@@ -733,6 +1284,198 @@ class Payload:detail["failures"] = failuresreturn self.record("tool_calling", not failures, detail)+ def assert_code_execution(self) -> bool:+ """The python tool must RUN, and the proof is a file on disk.++ `assert_tool_calling` above proves the model can EMIT a call. That is a+ different claim: a weather tool is never executed by Studio at all, the+ caller is expected to run it. The local `python` tool is executed by+ Studio itself, in a per-session sandbox, and the interesting failure is+ the loop offering the tool and never running it -- which looks+ identical from the response text, because the model will happily+ narrate a result it never received.++ So the evidence is not the prose. A token is minted here, the model is+ asked to write it to a file, and this reads it back out of+ `<studio home>/sandbox`, where `sandbox_root()` puts the per-session+ working directories. Nothing in the reply can fake that; only an+ executed `open(...).write(...)` puts those bytes on this disk.++ Two settings are not incidental and must not be "simplified":++ * `permission_mode = "off"`. `routes/inference.py` REJECTS a local+ python/terminal tool under `ask`, and under `auto` or an omitted+ default, with a 400 -- there is no confirmation channel here. The+ run would fail on configuration and look like a broken tool.+ * `tool_choice` is left alone. Forcing it would prove the schema is+ reachable, not that the loop runs what it selected, and the file is+ the claim either way.++ The filename is not asserted, only the CONTENT: a small local model+ rewording a path is not a Studio defect, and any file carrying the+ token was written by code that ran.+ """+ failures: list[str] = []+ detail: dict = {}++ token = "unslothcodeexec" + secrets_module.token_hex(8)+ sandbox = self.studio_home / "sandbox"+ detail["sandbox_root"] = str(sandbox)+ before = set(sandbox.rglob("*")) if sandbox.exists() else set()++ code, payload = self.chat(+ [+ {+ "role": "user",+ "content": (+ "Use the python tool to run exactly this code, then reply "+ "with the single word done:\n\n"+ f"open({token + '.txt'!r}, 'w').write({token!r})"+ ),+ }+ ],+ enable_tools = True,+ enabled_tools = ["python"],+ permission_mode = "off",+ max_tokens = 512,+ )+ detail["http_status"] = code+ if code != 200 or not isinstance(payload, dict):+ failures.append(+ f"the code-execution completion returned HTTP {code}: {str(payload)[:300]}"+ )+ else:+ choice = (payload.get("choices") or [{}])[0]+ message = choice.get("message") or {}+ detail["finish_reason"] = choice.get("finish_reason")+ detail["reply"] = (message.get("content") or "")[:300]++ # The claim, read off the filesystem rather than off the reply.+ written = []+ if sandbox.exists():+ for path in sandbox.rglob("*"):+ if not path.is_file():+ continue+ try:+ body = path.read_text(encoding = "utf-8", errors = "replace")+ except OSError:+ continue+ if token in body or token in path.name:+ written.append(str(path.relative_to(sandbox)))+ detail["files_carrying_the_token"] = written+ detail["new_sandbox_entries"] = sorted(+ str(p.relative_to(sandbox))+ for p in ((set(sandbox.rglob("*")) - before) if sandbox.exists() else set())+ )[:20]++ if not written:+ failures.append(+ "no file under the sandbox carries the token, so the python "+ "tool was offered but never executed -- whatever the reply says"+ )++ detail["failures"] = failures+ return self.record("code_execution", not failures, detail)++ def assert_web_search(self) -> bool:+ """The web_search tool must be EXECUTED, and the log is where that shows.++ Same shape as `assert_code_execution` and a different instrument,+ because a web search leaves nothing on disk to read back. Studio logs+ `execute_tool: name=...` from INSIDE `execute_tool`, so the line is+ emitted by execution rather than by selection -- a loop that offered+ the tool and never ran it produces no such line, which is the failure+ worth catching.++ What is deliberately NOT asserted: that the search returned results.+ `_web_search` fans out through ddgs with no API key, and a provider+ rate-limiting a Kaggle egress IP is a fact about the day, not a Studio+ defect. Failing on it would put a red in front of every PR for+ something no reader could act on. The reply and the result count are+ recorded so a human can see which happened.+ """+ failures: list[str] = []+ detail: dict = {}++ # TWO attempts, and the second is what makes a failure diagnosable.+ # `enabled_tools = ["web_search"]` is one name out of ALL_TOOLS, and+ # `routes/inference.py` also reads a request naming only hosted-tool+ # names as a provider-hosted ask. Omitting `enabled_tools` selects+ # every local tool instead, which is a different path through the same+ # loop. If the first fails and the second executes, the fault is in the+ # single-name selection; if neither does, the model will not call the+ # tool however it is offered. Reporting "the loop offered web_search+ # and never ran it" off one attempt was a guess dressed as a finding:+ # nothing in that run showed the tool had been offered at all.+ marker = "execute_tool: name=web_search"+ prompt = (+ "Search the web for the current version of the Linux kernel, "+ "then answer in one sentence."+ )+ attempts: list[dict] = []+ for label, selection in (+ ("named", {"enabled_tools": ["web_search"]}),+ ("all_local_tools", {}),+ ):+ before = ""+ try:+ before = self.server_log.read_text(encoding = "utf-8", errors = "replace")+ except OSError:+ pass++ code, payload = self.chat(+ [{"role": "user", "content": prompt}],+ enable_tools = True,+ permission_mode = "off",+ # Forced BY NAME. `tool_choice: "required"` was tried first and+ # changed nothing -- kernel unsloth-probe-studio-r3-0b85d4+ # returned the same parametric answer, "The current version of+ # the Linux kernel is 6.10", with executions 0. A bare+ # "required" says only that some tool must be called; the dict+ # form pins the function, and+ # `chat_template_helpers.forced_tool_name` reads exactly this+ # shape. Without a force this measures whether a 2B model+ # DECIDES to search, which is a model property rather than a+ # Studio one.+ tool_choice = {"type": "function", "function": {"name": "web_search"}},+ max_tokens = 512,+ **selection,+ )+ record = {"selection": label, "http_status": code}+ if code != 200 or not isinstance(payload, dict):+ record["error"] = str(payload)[:200]+ else:+ choice = (payload.get("choices") or [{}])[0]+ record["finish_reason"] = choice.get("finish_reason")+ record["reply"] = ((choice.get("message") or {}).get("content") or "")[:200]++ try:+ after = self.server_log.read_text(encoding = "utf-8", errors = "replace")+ except OSError:+ after = ""+ # Only what THIS request wrote, so an earlier attempt's tool call+ # cannot be read as this one's evidence.+ fresh = after[len(before) :]+ record["executions"] = fresh.count(marker)+ # Any tool at all, which separates "the loop ran and chose+ # something else" from "the loop never ran".+ record["any_tool_executions"] = fresh.count("execute_tool: name=")+ attempts.append(record)+ if record["executions"]:+ break++ detail["attempts"] = attempts+ detail["executions"] = sum(a["executions"] for a in attempts)+ if not detail["executions"]:+ failures.append(+ f"{marker!r} never appeared in the server log for either "+ f"selection, so web_search was not executed however it was "+ f"offered: {attempts}"+ )++ detail["failures"] = failures+ return self.record("web_search", not failures, detail)+# ----------------------------------------------------------- assertion Bdef assert_training(self) -> bool:@@ -1063,8 +1806,664 @@ class Payload:detail["failures"] = failuresreturn self.record("gguf_export", not failures, detail)+ # Every tab in Studio, and the backing endpoint its first render calls.+ # A tab whose router failed to import does not render a broken panel; the+ # route is simply absent and the request 404s, which is what this looks for.+ # Named by tab so a failure says which one, and kept as a mapping so a new+ # tab that is not covered here is visible as an omission rather than+ # invisible.+ TAB_ENDPOINTS = (+ # NOT /jobs/current: that legitimately 404s when no job is running,+ # so it would be red on correct behaviour. This one is unconditional+ # and lives in the same router package, so an import failure takes it+ # down with everything else.+ ("data_designer", "/api/data-recipe/seed/github/env-token"),+ ("image_creation", "/api/inference/images/status"),+ ("image_creation_gallery", "/api/inference/images/gallery"),+ ("video_creation", "/api/inference/video/status"),+ ("video_creation_gallery", "/api/inference/video/gallery"),+ ("image_training", "/api/train/diffusion/status"),+ ("image_training_runs", "/api/train/diffusion/runs"),+ ("training", "/api/train/status"),+ ("models", "/api/models/local"),+ ("datasets", "/api/datasets/local"),+ )++ def assert_tabs(self) -> bool:+ """Every tab's backing endpoint answers.++ This is a smoke and it is honest about being one: it does not click+ through the UI, it asks each tab's own first request. What it catches+ is the failure that actually happens -- a route module that raised on+ import, so the router was never mounted and the tab renders an error+ the moment it is opened. That is invisible to every other assertion+ here, all of which touch inference, training and export only.++ A 404 is the specific signal, so it is separated from other statuses+ in the record rather than folded into "not 200": a 500 is a live route+ with a broken handler and a 404 is a route that does not exist, and+ they lead a reader to different places.++ An endpoint answering 200 with a non-object is a failure too. Several+ of these are declared with a `response_model`, so a bare string or null+ means something upstream is substituting for the real handler.+ """+ failures: list[str] = []+ detail: dict = {"checked": len(self.TAB_ENDPOINTS)}+ results: dict = {}++ for name, path in self.TAB_ENDPOINTS:+ try:+ code, body = self.studio.get(path)+ except BaseException as exc: # noqa: BLE001+ results[name] = {"path": path, "error": f"{type(exc).__name__}: {exc}"[:200]}+ failures.append(f"{name}: {path} raised {type(exc).__name__}")+ continue+ entry = {"path": path, "status": code, "type": type(body).__name__}+ results[name] = entry+ if code == 404:+ failures.append(+ f"{name}: {path} is 404, so its router was never mounted -- "+ f"the tab renders an error the moment it is opened"+ )+ elif code >= 400:+ failures.append(f"{name}: {path} returned HTTP {code}: {str(body)[:200]}")+ elif not isinstance(body, (dict, list)):+ failures.append(+ f"{name}: {path} answered 200 with {type(body).__name__}, not a "+ f"JSON object, so something is standing in for the handler"+ )++ detail["tabs"] = results+ detail["failures"] = failures+ return self.record("tabs", not failures, detail)++ def assert_lora_vs_base(self, gguf: str | None) -> bool:+ """The exported model must answer DIFFERENTLY from the base it came from.++ This is the comparison an export check cannot make on its own. The GGUF+ assertion proves a file was produced, carries the magic, loads on the+ GPU and generates -- and every one of those is true of an export that+ silently merged nothing and shipped the base weights. A no-op merge is+ the regression here, and it is invisible to file size, to the magic and+ to "it generated text".++ Greedy decoding at temperature 0 makes it visible: identical weights+ answer identically, so a difference is the adapter.++ **The determinism control is not optional and comes first.** If the+ SAME weights, loaded twice, do not reproduce their own answer, then a+ difference between two models says nothing, and this reports that it+ could not compare rather than passing on the noise. Two loads of the+ base, then one of the export: the claim is only made once the+ instrument has been shown to be steady.++ The canary is REPORTED rather than asserted. Studio's training run is a+ handful of steps and whether that is enough to learn a specific string+ is a property of the run length, not of the export path -- asserting it+ would be a red about training tuning wearing an export label.+ """+ failures: list[str] = []+ detail: dict = {"gguf": gguf}+ prompt = [{"role": "user", "content": "What is the Unsloth Studio Kaggle canary?"}]++ if not gguf:+ failures.append(+ "no exported GGUF to compare: the export assertion did not "+ "produce one, so there is nothing to hold against the base"+ )+ detail["failures"] = failures+ return self.record("lora_vs_base", False, detail)++ def _say(model: str, variant, label: str) -> tuple[str | None, list]:+ loaded = self.load_model(model, variant = variant, label = label)+ if loaded["failures"]:+ return None, [f"{label} did not load: {loaded['failures'][0]}"]+ code, payload = self.chat(prompt)+ if code != 200 or not isinstance(payload, dict):+ return None, [f"{label} returned HTTP {code} on generation"]+ choices = payload.get("choices") or [{}]+ return ((choices[0].get("message") or {}).get("content") or ""), []++ base_one, problems = _say(self.args.chat_model, self.args.chat_variant, "base")+ failures += problems+ base_two, problems = _say(self.args.chat_model, self.args.chat_variant, "base_again")+ failures += problems+ detail["base_first"] = (base_one or "")[:200]+ detail["base_second"] = (base_two or "")[:200]++ if base_one is not None and base_two is not None:+ detail["base_reproduces_itself"] = base_one == base_two+ if base_one != base_two:+ failures.append(+ "the base model did not reproduce its own greedy answer "+ "across two loads, so a difference against the export "+ "would be noise rather than the adapter"+ )++ if not failures:+ tuned, problems = _say(gguf, None, "exported")+ failures += problems+ detail["exported_said"] = (tuned or "")[:200]+ detail["canary_in_exported"] = CANARY in (tuned or "")+ detail["canary_in_base"] = CANARY in (base_one or "")+ if tuned is not None:+ detail["differs_from_base"] = tuned != base_one+ if tuned == base_one:+ failures.append(+ "the exported model gave the base model's answer "+ "verbatim, so the adapter reached neither the merge "+ "nor the GGUF -- which every other export check passes"+ )++ detail["failures"] = failures+ return self.record("lora_vs_base", not failures, detail)++ def assert_image_generation(self) -> bool:+ """The image tab, end to end: load, generate, and download the PNG.++ Last priority and the smallest possible run -- 256x256 (the schema's+ floor) at 2 steps -- because the claim is that the path executes, not+ that the picture is good.++ "Nothing errored" is not the check. A diffusion pipeline that fails+ mid-way still writes a gallery record and still returns 200, and a+ pipeline whose weights never loaded produces a FLAT image, which is a+ valid PNG. So the evidence is the file:++ * the bytes start with the PNG magic, so what the download endpoint+ serves is really a PNG rather than a JSON error with a 200 on it;+ * the IHDR chunk says 256x256, read out of the header rather than+ taken from the gallery record -- the record repeats what was asked+ for, and the file says what was made;+ * the image is not one flat colour. A pipeline that produced nothing+ returns a uniform frame, which compresses to almost nothing, so this+ is checked on the decoded extrema where PIL is available and on a+ compressed-size floor where it is not. Both are recorded, so a+ reader can see which one ruled.+ """+ import urllib.error+ import urllib.request++ failures: list[str] = []+ detail: dict = {"model": self.args.image_model}+ want = 256++ try:+ code, body = self.studio.post(+ "/api/inference/images/load",+ {"model_path": self.args.image_model},+ timeout = self.args.export_timeout,+ )+ detail["load_status"] = code+ if code >= 400:+ failures.append(f"images/load returned HTTP {code}: {str(body)[:300]}")+ detail["failures"] = failures+ return self.record("image_generation", False, detail)++ # The load is ASYNCHRONOUS. It answers 200 having only accepted the+ # request, and generating against it answers 409 "No diffusion+ # model is loaded." -- measured on kernel+ # unsloth-probe-studio-r3-0b85d4, where load_status was 200 and+ # generate_status was 409 twelve lines later. So wait for+ # `images/status` to say `loaded`, and carry `load-progress` while+ # waiting: a download that stalls or errors is then reported as+ # what it is instead of arriving as a generation failure.+ deadline = time.time() + self.args.export_deadline+ status: dict = {}+ progress: dict = {}+ while time.time() < deadline:+ _, status_body = self.studio.get("/api/inference/images/status")+ status = status_body if isinstance(status_body, dict) else {}+ if status.get("loaded"):+ break+ _, progress_body = self.studio.get("/api/inference/images/load-progress")+ progress = progress_body if isinstance(progress_body, dict) else {}+ if progress.get("phase") == "error":+ break+ time.sleep(5.0)+ detail["load_status_body"] = {+ k: status.get(k) for k in ("loaded", "repo_id", "family", "device", "model_kind")+ }+ detail["load_progress"] = {k: progress.get(k) for k in ("phase", "fraction", "error")}+ if not status.get("loaded"):+ failures.append(+ f"the diffusion model never reported loaded within "+ f"{self.args.export_deadline}s: status={detail['load_status_body']} "+ f"progress={detail['load_progress']}"+ )+ detail["failures"] = failures+ return self.record("image_generation", False, detail)++ code, body = self.studio.post(+ "/api/inference/images/generate",+ {+ "prompt": "a red square on a blue background",+ "width": want,+ "height": want,+ "steps": 2,+ "guidance": 0.0,+ "seed": 3407,+ },+ timeout = self.args.export_timeout,+ )+ detail["generate_status"] = code+ images = (body or {}).get("images") if isinstance(body, dict) else None+ detail["generated_count"] = len(images or [])+ if code >= 400 or not images:+ failures.append(+ f"images/generate returned HTTP {code} with {len(images or [])} "+ f"images: {str(body)[:300]}"+ )+ detail["failures"] = failures+ return self.record("image_generation", False, detail)++ record = images[0]+ detail["record"] = {k: record.get(k) for k in ("id", "width", "height", "steps")}+ image_id = record.get("id")++ # The PNG itself, fetched raw. The JSON client decodes to utf-8,+ # which would corrupt the bytes the whole check is about.+ request = urllib.request.Request(+ f"http://127.0.0.1:{self.args.port}/api/inference/images/gallery/{image_id}/file"+ )+ if self.studio.token:+ request.add_header("Authorization", f"Bearer {self.studio.token}")+ with urllib.request.urlopen(request, timeout = 120) as response:+ png = response.read()+ detail["content_type"] = response.headers.get("Content-Type")+ detail["png_bytes"] = len(png)++ if not png.startswith(b"\x89PNG\r\n\x1a\n"):+ failures.append(+ f"the download did not start with the PNG magic, so what it "+ f"served is not a PNG: {png[:16]!r}"+ )+ else:+ # IHDR is fixed-offset: 8 magic, 4 length, 4 type, then width+ # and height as big-endian uint32.+ width = int.from_bytes(png[16:20], "big")+ height = int.from_bytes(png[20:24], "big")+ detail["png_size"] = [width, height]+ if (width, height) != (want, want):+ failures.append(+ f"the PNG header says {width}x{height}, not {want}x{want} -- "+ f"the file is not what was asked for, whatever the record says"+ )++ flat = None+ try:+ from PIL import Image # noqa: PLC0415++ with Image.open(io.BytesIO(png)) as opened:+ bands = opened.convert("RGB").getextrema()+ detail["extrema"] = bands+ flat = all(low == high for low, high in bands)+ detail["flatness_source"] = "pixels"+ except Exception as exc: # noqa: BLE001+ detail["pil_error"] = f"{type(exc).__name__}: {exc}"[:200]+ # A uniform 256x256 frame compresses to well under 2 KB.+ flat = len(png) < 2000+ detail["flatness_source"] = "compressed size"+ if flat:+ failures.append(+ f"the image is one flat colour ({detail['flatness_source']}), "+ f"which is what a pipeline whose weights never loaded "+ f"produces -- and it is a perfectly valid PNG"+ )+ except BaseException as exc: # noqa: BLE001+ failures.append(f"the image path raised: {type(exc).__name__}: {exc}"[:300])+ finally:+ # Always, or a diffusion pipeline holds the card for whatever runs+ # next and that failure lands on the wrong assertion.+ try:+ self.studio.post("/api/inference/images/unload", {}, timeout = 120)+ except BaseException: # noqa: BLE001+ pass++ detail["failures"] = failures+ return self.record("image_generation", not failures, detail)++ def assert_cloudflare(self) -> bool:+ """`unsloth run --cloudflare`: a public URL that serves, and refuses.++ This is the only assertion in the payload that reaches the public+ internet, so what it claims is deliberately narrow and what it refuses+ to claim is stated:++ 1. cloudflared is fetched and a quick tunnel is established, and the+ URL printed is a real `*.trycloudflare.com` host rather than the+ `api.trycloudflare.com` that appears in cloudflared's own FAILURE+ lines -- that is a live trap, and Studio's own regex carries the+ same negative lookahead for it;+ 2. the tunnel SERVES: `/api/health` answers through the public URL,+ which is what separates "a URL was printed" from "a URL that works";+ 3. the tunnel REFUSES an unauthenticated request. A public URL onto a+ CI machine is only defensible if it is behind auth, and this is the+ check that says so rather than assuming it.++ `--host 0.0.0.0` is not incidental. Studio raises a quick tunnel for+ WILDCARD binds; on 127.0.0.1 there is nothing to publish and no URL is+ printed, which would read as a broken feature.++ A tunnel that cannot be established AT ALL is reported rather than+ failed, and the reason is carried from the log. Kaggle egress to+ cloudflared's release host is not something this repo controls, and the+ directive asks for this "if possible". The narrowness is enforced: the+ excuse applies only when NO url was printed, and never once one was.+ """+ failures: list[str] = []+ detail: dict = {}+ port = self.args.port + 2+ log_path = self.outdir / "unsloth_cloudflare.log"+ detail["port"] = port++ head = self.studio_command()[:1] or [sys.executable]+ if head[0] == sys.executable:+ head = [sys.executable, "-c", "from unsloth_cli import app; app()"]+ cmd = head + [+ "run",+ "--model",+ self.args.chat_model,+ "--port",+ str(port),+ # WILDCARD. A loopback bind publishes nothing and prints no URL.+ "--host",+ "0.0.0.0",+ "--api-only",+ "--cloudflare",+ "--start-api-key-marker",+ "--max-seq-length",+ str(self.args.studio_ctx),+ ]+ if self.args.chat_variant:+ cmd += ["--gguf-variant", self.args.chat_variant]+ detail["command"] = " ".join(cmd)++ env = dict(os.environ)+ env["UNSLOTH_STUDIO_HOME"] = str(self.studio_home)+ env.setdefault("HF_HOME", str(self.studio_home / "cache" / "huggingface"))+ env["PYTHONUNBUFFERED"] = "1"+ env["UNSLOTH_DISABLE_STATISTICS"] = "1"++ handle = open(log_path, "ab")+ proc = subprocess.Popen(+ cmd,+ cwd = str(self.repo_root),+ env = env,+ stdout = handle,+ stderr = subprocess.STDOUT,+ )++ api_key = None+ url = None+ try:+ deadline = time.time() + self.args.health_deadline+ while time.time() < deadline:+ if proc.poll() is not None:+ failures.append(+ f"`unsloth run --cloudflare` exited with code {proc.returncode}"+ )+ break+ text = log_path.read_text(encoding = "utf-8", errors = "replace")+ if api_key is None and "UNSLOTH_START_API_KEY:" in text:+ api_key = text.split("UNSLOTH_START_API_KEY:", 1)[1].split("\n", 1)[0].strip()+ if api_key:+ self.secrets.add(api_key)+ if url is None:+ # The SAME negative lookahead Studio's own matcher uses.+ # `api.trycloudflare.com` appears in cloudflared's failure+ # lines and is never a usable tunnel.+ found = re.search(r"https://(?!api\.)[A-Za-z0-9-]+\.trycloudflare\.com", text)+ if found:+ url = found.group(0)+ if url and api_key:+ break+ time.sleep(2.0)++ detail["tunnel_url_seen"] = bool(url)+ if url is None:+ # Reported, not failed, and ONLY here: nothing was published,+ # so there is nothing to have gone wrong with. The reason comes+ # off the log rather than being assumed.+ tail = log_path.read_text(encoding = "utf-8", errors = "replace")[-600:]+ detail["no_tunnel_reason"] = self.scrub(tail)+ detail["reported_not_failed"] = True+ else:+ # The host is a secret in the sense that matters here: it is a+ # live public route to this machine, and the artifact is read+ # by people who are not running it.+ self.secrets.add(url)+ public = Studio(url, timeout = 30.0)+ code, body = public.get("/api/health", auth = False)+ detail["public_health_status"] = code+ # A tunnel that resolves but serves Cloudflare's own error page+ # answers 530, which is a URL that does not work.+ if code != 200:+ failures.append(+ f"the quick tunnel URL answered HTTP {code} on /api/health, "+ f"so a URL was published that does not serve"+ )++ # And it must REFUSE. A public URL onto a CI box is only+ # defensible behind auth, so this is asserted, not assumed.+ code, _ = public.post(+ "/v1/chat/completions",+ {+ "model": "default",+ "messages": [{"role": "user", "content": "hi"}],+ "max_tokens": 8,+ },+ auth = False,+ )+ detail["public_unauthenticated_status"] = code+ if code < 400:+ failures.append(+ f"an UNAUTHENTICATED request through the public tunnel "+ f"was accepted ({code}), so the quick tunnel exposes "+ f"this machine's inference to anyone with the URL"+ )+ except BaseException as exc: # noqa: BLE001+ failures.append(f"driving the tunnel raised: {type(exc).__name__}: {exc}"[:300])+ finally:+ proc.terminate()+ try:+ proc.wait(timeout = 60)+ except subprocess.TimeoutExpired:+ proc.kill()+ handle.close()++ detail["failures"] = failures+ return self.record("cloudflare", not failures, detail)+# ------------------------------------------------------ existing drivers+ def assert_cli_run(self) -> bool:+ """`unsloth run`: a model server started from the CLI, driven by its key.++ This is the headless path a user scripts, and nothing in CI covers it.+ It is a different launch from `unsloth studio`: `run` starts the+ backend, waits for health, mints an API key IN-PROCESS, and then loads+ the model over HTTP. Any of those four can break without the others+ noticing, and the command still prints a banner.++ Four claims, and each rules out a way the previous one passes hollow:++ 1. the server becomes healthy on the port it was given;+ 2. the model reaches the GPU -- measured as device VRAM growth across+ the launch, not as a line in the banner. `--api-only` on a card the+ chat-UI phase has already emptied makes that delta this launch's;+ 3. the key the command minted AUTHENTICATES a real completion, and the+ completion is non-empty. A key that is printed and rejected is+ worse than no key, because the failure surfaces in a user's+ integration rather than here;+ 4. a CORRUPTED key is refused. Without it, a server ignoring the header+ entirely passes claim 3.++ `--start-api-key-marker` is how the key is obtained: it prints+ `UNSLOTH_START_API_KEY: <key>`, which is the mechanism `unsloth start`+ itself uses. The value is registered as a secret the moment it is read,+ before anything scrubs a log on the way into the evidence bundle.+ """+ failures: list[str] = []+ detail: dict = {}+ port = self.args.port + 1+ log_path = self.outdir / "unsloth_run.log"+ detail["port"] = port++ baseline = nvidia_used_mib()+ detail["vram_before_mib"] = baseline+ # The per-process reading is taken alongside the device one because the+ # device one is only valid when this payload owns the card, and under+ # --studio-concurrent it does not. See the verdict below.+ apps_before = nvidia_compute_apps() or {}+ detail["compute_apps_before"] = apps_before++ head = self.studio_command()[:1] or [sys.executable]+ if head[0] == sys.executable:+ head = [sys.executable, "-c", "from unsloth_cli import app; app()"]+ cmd = head + [+ "run",+ "--model",+ self.args.chat_model,+ "--port",+ str(port),+ "--host",+ "127.0.0.1",+ # Headless: no UI to serve and no browser to open, which is the+ # shape this path is for.+ "--api-only",+ # Never a public URL from CI. --secure would imply one.+ "--no-cloudflare",+ "--start-api-key-marker",+ "--max-seq-length",+ str(self.args.studio_ctx),+ ]+ if self.args.chat_variant:+ cmd += ["--gguf-variant", self.args.chat_variant]+ detail["command"] = " ".join(cmd)++ env = dict(os.environ)+ env["UNSLOTH_STUDIO_HOME"] = str(self.studio_home)+ env.setdefault("HF_HOME", str(self.studio_home / "cache" / "huggingface"))+ env["PYTHONUNBUFFERED"] = "1"+ env["UNSLOTH_DISABLE_STATISTICS"] = "1"++ handle = open(log_path, "ab")+ proc = subprocess.Popen(+ cmd,+ cwd = str(self.repo_root),+ env = env,+ stdout = handle,+ stderr = subprocess.STDOUT,+ )++ api_key = None+ client = Studio(f"http://127.0.0.1:{port}")+ try:+ deadline = time.time() + self.args.health_deadline+ while time.time() < deadline:+ if proc.poll() is not None:+ failures.append(f"`unsloth run` exited early with code {proc.returncode}")+ break+ text = log_path.read_text(encoding = "utf-8", errors = "replace")+ if api_key is None and "UNSLOTH_START_API_KEY:" in text:+ api_key = text.split("UNSLOTH_START_API_KEY:", 1)[1].split("\n", 1)[0].strip()+ if api_key:+ # Before anything else reads this file.+ self.secrets.add(api_key)+ if api_key and health_is_ready(client.get("/api/health", auth = False)[1]):+ break+ time.sleep(2.0)++ detail["saw_api_key"] = bool(api_key)+ if not api_key:+ failures.append(+ "`unsloth run` never printed UNSLOTH_START_API_KEY, so it "+ "did not reach the point where it mints a key"+ )++ if api_key:+ client.token = api_key+ code, body = client.post(+ "/v1/chat/completions",+ {+ "model": "default",+ "messages": [{"role": "user", "content": "Say hello in one word."}],+ "max_tokens": 32,+ "temperature": 0.0,+ },+ timeout = self.args.chat_timeout,+ )+ detail["completion_status"] = code+ text = ""+ if code == 200 and isinstance(body, dict):+ text = ((body.get("choices") or [{}])[0].get("message") or {}).get(+ "content"+ ) or ""+ detail["generated"] = text[:200]+ if code != 200:+ failures.append(+ f"the key `unsloth run` minted did not authenticate a "+ f"completion: HTTP {code} {str(body)[:200]}"+ )+ elif not text.strip():+ failures.append("the CLI-served model returned empty content")++ # And a corrupted key must be refused, or the check above+ # passes on a server that ignores the header.+ client.token = api_key[:-4] + "0000" if len(api_key) > 8 else "bogus"+ code, _ = client.post(+ "/v1/chat/completions",+ {+ "model": "default",+ "messages": [{"role": "user", "content": "hi"}],+ "max_tokens": 8,+ },+ timeout = self.args.chat_timeout,+ )+ detail["bad_key_status"] = code+ if code < 400:+ failures.append(+ f"a corrupted API key was accepted ({code}), so the "+ f"check above passes whatever is sent"+ )++ # AFTER a completion has come back, not after the API-key banner.+ # `unsloth run` prints the key while it is still starting, and on+ # kernel unsloth-probe-studio-full2-815a0c the sample landed before+ # llama-server had the weights anywhere: 0.0 MiB of growth on a+ # launch whose own log says+ # `Starting llama-server: ... -ngl -1 --fit off`, which is Studio+ # asking for every layer on the card. A served completion is the+ # only cheap proof the weights are resident, so the ruler goes+ # after it.+ settled = nvidia_used_mib()+ detail["vram_after_mib"] = settled+ apps_after = nvidia_compute_apps()+ detail["compute_apps"] = apps_after+ failure, measured = cli_run_gpu_failure(+ apps_before,+ apps_after,+ baseline,+ settled,+ )+ detail.update(measured)+ if failure:+ failures.append(failure)+ except BaseException as exc: # noqa: BLE001+ failures.append(f"driving `unsloth run` raised: {type(exc).__name__}: {exc}"[:300])+ finally:+ proc.terminate()+ try:+ proc.wait(timeout = 60)+ except subprocess.TimeoutExpired:+ proc.kill()+ handle.close()++ detail["failures"] = failures+ return self.record("cli_run", not failures, detail)+def assert_chat_ui(self) -> bool:"""Drive the repo's own chat UI driver. Runs last: it stops the server."""driver = self.repo_root / "tests" / "studio" / "playwright_chat_ui.py"@@ -1189,7 +2588,16 @@ class Payload:def _pack(*, with_screenshots: bool, log_tail_bytes: int | None = None) -> bytes:buf = io.BytesIO()with tarfile.open(fileobj = buf, mode = "w:gz") as tar:- for name in ("studio_gpu_report.json", "playwright_chat_ui.log", "studio.log"):+ for name in (+ "studio_gpu_report.json",+ "unsloth_cloudflare.log",+ "playwright_chat_ui.log",+ "studio.log",+ # `unsloth run` prints the API key it mints; redacted() is+ # what keeps it out of the artifact, and a file nobody+ # packs is a file nobody redacts either.+ "unsloth_run.log",+ ):path = self.outdir / nameif path.is_file():scrubbed = self.redacted(path)@@ -1246,9 +2654,21 @@ class Payload:if not self.authenticate():return self.finish()+ # Before the GPU work: it needs nothing but a logged-in session, and+ # putting it after a 20-minute training run would mean a training+ # failure hides whether API keys work at all.+ self.assert_api_key()++ # Before any model work: these are pure route checks that need only a+ # logged-in session, and putting them after a 20-minute training run+ # would mean a training failure hides whether the tabs exist at all.+ self.assert_tabs()+gpu_ok = self.assert_gpu_inference()if gpu_ok:self.assert_tool_calling()+ self.assert_code_execution()+ self.assert_web_search()else:# Tool calling on a CPU fallback would be a green tick for a# question nobody asked. Skip it and say so.@@ -1257,6 +2677,48 @@ class Payload:False,{"failures": ["skipped: the model was not on the GPU, so this proves nothing"]},)+ self.record(+ "code_execution",+ False,+ {"failures": ["skipped: the model was not on the GPU, so this proves nothing"]},+ )+ self.record(+ "web_search",+ False,+ {"failures": ["skipped: the model was not on the GPU, so this proves nothing"]},+ )++ # AFTER the GPU inference assertions and BEFORE training: it reloads+ # the chat model with different flags, so running it earlier would+ # change the model the inference checks measured, and running it after+ # training would put a reload between the adapter and the export.+ if gpu_ok:+ self.assert_server_flags()+ # AFTER the reload, because that is what pins the window to+ # --studio-ctx: a compaction check against an unknown context+ # length cannot say whether the prompt was over it.+ self.assert_compaction()+ else:+ self.record(+ "compaction",+ False,+ {+ "failures": [+ "skipped: the model was not on the GPU, so the window "+ "the check overflows was never pinned"+ ]+ },+ )+ self.record(+ "server_flags",+ False,+ {+ "failures": [+ "skipped: the model was not on the GPU, so a "+ "tensor_split check proves nothing"+ ]+ },+ )trained = self.assert_training()adapter_dir = None@@ -1265,8 +2727,47 @@ class Payload:adapter_dir = entry.get("output_dir")self.assert_gguf_export(adapter_dir if trained else None)+ # Straight after, and it reads the export's own recorded path: the+ # comparison is only meaningful against the file that assertion made.+ exported_gguf = None+ for entry in self.assertions:+ if entry["name"] == "gguf_export":+ exported_gguf = entry.get("gguf")+ self.assert_lora_vs_base(exported_gguf)++ # BEFORE the UI driver, because assert_chat_ui ends by stopping the+ # server and every request below it would then be refused at the+ # socket. Measured on kernel unsloth-probe-studio-full2-815a0c, where+ # this assertion reported `URLError: Connection refused` and read as a+ # broken image path on a server that had simply been shut down.+ # Placed after all the language work regardless: a diffusion pipeline+ # is the largest single thing this payload puts on a T4, and it is+ # unloaded in a finally either way.+ if self.args.image_generation:+ self.assert_image_generation()+if not self.args.skip_ui:self.assert_chat_ui()++ # LAST, and the order is the design. `unsloth run` starts a SECOND+ # backend against the same studio home, and two backends sharing one+ # home's state is not a configuration anybody ships. assert_chat_ui+ # ends by stopping the server, so by here the port is free, the card is+ # empty, and the VRAM delta below measures this launch alone.+ self.assert_cli_run()++ # LAST of all, and the only thing here that touches the public+ # internet. Its own launch rather than a flag on the one above,+ # because the tunnel needs a WILDCARD bind and assert_cli_run's claim+ # is about a loopback server.+ if self.args.cloudflare_check:+ self.assert_cloudflare()+ else:+ self.record(+ "cloudflare",+ False,+ {"failures": ["skipped: --no-cloudflare-check was passed"]},+ )return self.finish()def finish(self) -> int:@@ -1298,9 +2799,54 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:ap.add_argument("--repo-root", required = True, help = "the unsloth checkout under test")ap.add_argument("--studio-home", required = True, help = "UNSLOTH_STUDIO_HOME for this run")ap.add_argument("--port", type = int, default = 18902)- ap.add_argument("--chat-model", default = "unsloth/Qwen3.5-2B-GGUF")+ ap.add_argument(+ # 2048 constrains the runs, as the brief asks. It also keeps the+ # KV-cache check about the cache: an unconstrained context on a 14.56GB+ # card turns it into an OOM about something else.+ "--studio-ctx",+ dest = "studio_ctx",+ type = int,+ default = 2048,+ help = "context length to pin the llama.cpp server to",+ )+ ap.add_argument(+ "--image-model",+ dest = "image_model",+ default = "unsloth/sdxl-turbo",+ help = "diffusion repo for the image-generation assertion",+ )+ ap.add_argument(+ # OFF by default: it is the last-priority item and it pulls a diffusion+ # checkpoint the rest of the payload has no use for. A dispatch that+ # wants it says so.+ "--image-generation",+ dest = "image_generation",+ action = "store_true",+ default = False,+ help = "load a diffusion model and generate one 256x256 image at 2 steps",+ )+ ap.add_argument(+ # On by default because the directive asks for it, and off-able because+ # it is the one assertion here that reaches the public internet.+ "--no-cloudflare-check",+ dest = "cloudflare_check",+ action = "store_false",+ default = True,+ help = "skip the quick-tunnel assertion (it opens a public URL)",+ )+ ap.add_argument(+ # Empty means "use the bootstrap password", which is the behaviour this+ # payload had before. A caller passes `auto` to have one generated.+ "--studio-password",+ default = "",+ help = "start Studio with --password and log in with it; 'auto' generates one",+ )+ # MTP-GGUF, not the plain GGUF: multi-token prediction is a distinct+ # serving path in llama.cpp, and a leg pointed at the plain repo cannot+ # tell whether it works.+ ap.add_argument("--chat-model", default = "unsloth/Qwen3.5-2B-MTP-GGUF")ap.add_argument("--chat-variant", default = "UD-Q4_K_XL")- ap.add_argument("--train-model", default = "unsloth/Qwen2.5-0.5B-Instruct")+ ap.add_argument("--train-model", default = "unsloth/Qwen3.5-2B")ap.add_argument("--max-steps", type = int, default = 8)ap.add_argument("--quantization", default = "q8_0")ap.add_argument(diff --git a/tests/kaggle/studio_gpu/studio_client.py b/tests/kaggle/studio_gpu/studio_client.pyindex daf5b019952..6320a2ca273 100644--- a/tests/kaggle/studio_gpu/studio_client.py+++ b/tests/kaggle/studio_gpu/studio_client.py@@ -368,9 +368,22 @@ def nonfinite_losses(status: Any) -> list:def newest_gguf(root: str | Path) -> Path | None:- """The most recently written ``.gguf`` under ``root``, if any."""+ """The most recently written MODEL ``.gguf`` under ``root``, if any.++ ``mmproj`` sidecars are excluded, and that is not a tidy-up. A vision export+ writes two files -- ``Qwen3.5-2B.Q8_0.gguf`` and+ ``Qwen3.5-2B.F16-mmproj.gguf`` -- and the projector is often the newer of+ the two. Handing it to llama.cpp as a model is not an error: the server+ starts, reports ``gpu_layers=-1``, offloads nothing and still returns text,+ so the run reads as a GPU failure in the export assertion. Measured on+ kernel unsloth-probe-studio-full2-815a0c, where exactly that happened.+ """root = Path(root)if not root.is_dir():return None- candidates = sorted(root.rglob("*.gguf"), key = lambda p: p.stat().st_mtime, reverse = True)+ candidates = sorted(+ (p for p in root.rglob("*.gguf") if "mmproj" not in p.name.lower()),+ key = lambda p: p.stat().st_mtime,+ reverse = True,+ )return candidates[0] if candidates else Nonediff --git a/tests/kaggle/t4_smoke/gguf_export.py b/tests/kaggle/t4_smoke/gguf_export.pynew file mode 100644index 00000000000..1c57e884bb4--- /dev/null+++ b/tests/kaggle/t4_smoke/gguf_export.py@@ -0,0 +1,279 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""Export a trained model to GGUF, and prove the file exists and is usable.++Shared by the payloads, in the shape the rest of this directory uses: nothing+here raises, everything returns a record, and a separate pure function turns+that record into failures. A diagnostic that kills the payload it diagnoses+leaves the leg reporting nothing at all.++Every constant below is a measurement, not a guess. The two that a natural+implementation gets WRONG, both learned by getting them wrong:++**1. The GGUF is not in the directory you passed.** On kernel+unsloth-probe-gguf-q8-peft-920e3e, `save_pretrained_gguf("/tmp/q8p", ...)` put+the merged `model.safetensors` (1136.9 MB) in `/tmp/q8p` and the actual+`qwen3-0.6b.Q8_0.gguf` (609.8 MB) in the SIBLING `/tmp/q8p_gguf`. Code that+globs the directory it passed finds no `.gguf`, raises nothing, and reports a+successful export. So this searches both and records which one held the file.++**2. `install_llama_cpp()` returns a TUPLE**, `(llama-quantize,+convert_hf_to_gguf.py)`, not a path to a bin directory. An earlier probe+treated it as a directory, found nothing in it, and reported "0 binaries" --+which was the probe being wrong rather than the bundle being empty.++Two more facts worth having here rather than rediscovering:++* llama.cpp installs PREBUILT on Kaggle, about 10-46s, logging "Installing+ prebuilt llama.cpp ... - skipping compilation". A source build is silent,+ correct, and costs many minutes on 4 vCPUs, so "it worked" is not the+ interesting assertion -- "it did not compile" is.+* The bundle is the `-cpu` one, so GGUF inference runs on CPU. Fine for a 0.6B+ or a 2B; a reason not to lean on GGUF inference for a 20B.++A model that REFUSES the requested quantization is not a failure. gpt-oss+answers q8_0 with "GPT-OSS does not support GGUF quantization (requested:+q8_0). Overriding to MXFP4 format", by design, so the caller says what it will+accept rather than this module assuming q8_0 is universal.+"""++from __future__ import annotations++import os+import subprocess+import time++# Where a GGUF may legitimately appear, relative to the directory passed to+# save_pretrained_gguf. The empty string is that directory itself; "_gguf" is+# the sibling unsloth actually writes to. Both are searched because relying on+# either alone has already produced a wrong answer.+GGUF_SEARCH_SUFFIXES = ("", "_gguf")++# Marks the prebuilt branch in unsloth's own install output.+PREBUILT_MARKER = "skipping compilation"++# Substrings that mean a source build happened instead. If any appears, the+# leg took the expensive path and should say so even though it succeeded.+SOURCE_BUILD_MARKERS = ("cmake", "Building llama.cpp", "make -j")+++def find_ggufs(save_dir: str) -> list:+ """Every .gguf reachable from `save_dir`, with the directory that held it."""+ found = []+ for suffix in GGUF_SEARCH_SUFFIXES:+ candidate = save_dir + suffix+ if not os.path.isdir(candidate):+ continue+ for root, _dirs, files in os.walk(candidate):+ for name in files:+ if not name.endswith(".gguf"):+ continue+ path = os.path.join(root, name)+ found.append(+ {+ "path": path,+ "mb": round(os.path.getsize(path) / 1024**2, 1),+ "found_in": candidate,+ "suffix": suffix,+ }+ )+ return sorted(found, key = lambda f: -f["mb"])+++def export_gguf(+ model,+ tokenizer,+ save_dir: str,+ *,+ quantization: str = "q8_0",+) -> dict:+ """Export and report. Never raises."""+ record = {"save_dir": save_dir, "requested_quantization": quantization}++ started = time.time()+ try:+ model.save_pretrained_gguf(save_dir, tokenizer, quantization_method = quantization)+ record["ok"] = True+ except BaseException as exc: # noqa: BLE001+ # unsloth/save.py:4777 wraps the real cause in+ # `RuntimeError(f"Failed to save model: {e}")` and interpolates str(e)+ # from exceptions whose args are empty, producing a message that ends+ # at the colon with no cause in it. Record the type too, so a report+ # from that path still names something.+ record["ok"] = False+ record["error"] = f"{type(exc).__name__}: {exc}"[:4000]+ record["seconds"] = round(time.time() - started, 1)++ record["ggufs"] = find_ggufs(save_dir)+ return record+++def run_gguf(+ gguf_path: str,+ llama_cpp_dir: str,+ *,+ max_tokens: int = 16,+ timeout: int = 240,+) -> dict:+ """Run the exported file, because an existing GGUF is not a working one.++ NOT via `llama-cli`. It hung for its entire budget twice on Kaggle -- 600s+ on kernel unsloth-probe-gguf-q8-peft-920e3e and 180s on -gguf-infer-50a98a,+ the second with stdin closed AND `-no-cnv` -- while 16 tokens from a 610 MB+ Q8_0 on CPU is seconds of work. Whatever it waits for is not stdin, so this+ uses binaries that cannot be interactive at all: `llama-bench` runs a fixed+ workload and exits, `llama-completion` is one-shot.+ """+ record = {"gguf": gguf_path}+ for name, argv in (+ ("bench", ["llama-bench", "-m", gguf_path, "-p", "8", "-n", str(max_tokens), "-r", "1"]),+ (+ "completion",+ [+ "llama-completion",+ "-m",+ gguf_path,+ "-p",+ "The capital of France is",+ "-n",+ str(max_tokens),+ "--temp",+ "0",+ ],+ ),+ ):+ exe = os.path.join(llama_cpp_dir, argv[0])+ if not os.path.exists(exe):+ record[name] = {"skipped": f"no {argv[0]} in the bundle"}+ continue+ started = time.time()+ try:+ proc = subprocess.run(+ [exe] + argv[1:],+ capture_output = True,+ text = True,+ timeout = timeout,+ stdin = subprocess.DEVNULL,+ )+ record[name] = {+ "seconds": round(time.time() - started, 1),+ "returncode": proc.returncode,+ "stdout": proc.stdout[-4000:],+ "stderr": proc.stderr[-2000:],+ }+ except BaseException as exc: # noqa: BLE001+ record[name] = {+ "seconds": round(time.time() - started, 1),+ "error": f"{type(exc).__name__}: {exc}"[:1000],+ }+ # One success is enough; the second is only run when the first is+ # unavailable or failed, so a healthy leg pays for one process.+ if record[name].get("returncode") == 0:+ break+ return record+++def export_failures(record: dict, *, accept_quantizations = None) -> list:+ """The pass rule, as a pure function of the record.++ `accept_quantizations` is the set of quantization names the caller will+ accept in the filename. Callers pass more than one where the model is+ allowed to override the request -- gpt-oss answers q8_0 with MXFP4 by+ design, and failing on that would be failing on documented behaviour.+ """+ if not record:+ return ["GGUF export was never run"]++ failures = []+ if not record.get("ok"):+ failures.append("GGUF export raised: " + str(record.get("error", "no error recorded")))++ ggufs = record.get("ggufs") or []+ if not ggufs:+ # The trap, stated in the failure itself so the reader does not have to+ # know it: an export can "succeed" and leave no GGUF anywhere.+ failures.append(+ f"no .gguf under {record.get('save_dir')!r} or its _gguf sibling, "+ f"even though the export reported ok={record.get('ok')}"+ )+ return failures++ biggest = ggufs[0]+ if biggest["mb"] <= 1.0:+ failures.append(+ f"the largest .gguf is {biggest['mb']} MB ({biggest['path']}), which is "+ f"a header and no weights"+ )++ if accept_quantizations:+ names = [os.path.basename(g["path"]).lower() for g in ggufs]+ accepted = [q.lower() for q in accept_quantizations]+ if not any(q in n for n in names for q in accepted):+ failures.append(+ f"no exported file names any accepted quantization {sorted(accept_quantizations)}: "+ f"got {names}"+ )+ return failures+++def run_failures(record: dict) -> list:+ """Did the exported file actually produce output?"""+ if not record:+ return ["the exported GGUF was never run"]++ attempts = {k: v for k, v in record.items() if isinstance(v, dict)}+ if not attempts:+ return ["no runner was attempted against the exported GGUF"]+ if all(a.get("skipped") for a in attempts.values()):+ return [+ "every GGUF runner was missing from the llama.cpp bundle: "+ + ", ".join(sorted(attempts))+ ]+ if any(a.get("returncode") == 0 for a in attempts.values()):+ return []++ detail = "; ".join(+ f"{name}: "+ + (+ a.get("error")+ or a.get("skipped")+ or f"rc={a.get('returncode')} {(a.get('stderr') or '')[-200:]}"+ )+ for name, a in sorted(attempts.items())+ )+ return [f"the exported GGUF produced no output from any runner ({detail})"]+++def llama_cpp_facts(install_output: str, returned) -> dict:+ """Was the PREBUILT branch taken, and what did the installer hand back?++ `returned` is whatever `install_llama_cpp()` returned -- a TUPLE of+ (llama-quantize, convert_hf_to_gguf.py) as measured, not a directory. The+ directory is derived from it rather than assumed, so a release that changes+ the layout shows up as a missing binary instead of a wrong path.+ """+ paths = list(returned) if isinstance(returned, (tuple, list)) else [returned]+ paths = [str(p) for p in paths]+ return {+ "returned": paths,+ "all_exist": all(os.path.exists(p) for p in paths) if paths else False,+ "dir": os.path.dirname(paths[0]) if paths else None,+ # TRI-STATE, and the third state is the point. The prebuilt banner is+ # printed once, by the install that actually downloads the bundle; a+ # second cycle in the same session finds llama.cpp already there and+ # prints nothing. Measured on unsloth-probe-visleg-full-b3a317, where+ # cycle 0 read `prebuilt: true` and cycle 1 read `prebuilt: false` for+ # the SAME installation -- and `false` reads as "built from source",+ # which is the one thing this field exists to catch.+ #+ # None means "this run did not install it, so it cannot say". The+ # source-build markers stay a plain list, because those are printed by+ # the build itself and their absence is meaningful either way.+ "prebuilt": (+ True+ if PREBUILT_MARKER in (install_output or "")+ else (None if not (install_output or "").strip() else False)+ ),+ "source_build_markers": [m for m in SOURCE_BUILD_MARKERS if m in (install_output or "")],+ }diff --git a/tests/kaggle/t4_smoke/kernel_provenance.py b/tests/kaggle/t4_smoke/kernel_provenance.pynew file mode 100644index 00000000000..2e71b7646da--- /dev/null+++ b/tests/kaggle/t4_smoke/kernel_provenance.py@@ -0,0 +1,163 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""Which fast kernels a model actually got, and where each one came from.++**The failure this is built to catch is not "kernel missing".** It is "kernel+built from source", which is silent, correct, slow, and reports the same+`__version__` as the wheel. Only the module's `__file__` and the install log+tell them apart, so a version check here would pass while proving nothing.++**Read AFTER the model load, not before.** Measured on kernel+`unsloth-probe-vision-recon-c76ea3`: `fla` is not importable before+`from_pretrained` and IS importable after, because unsloth reaches for it lazily+when it sees the model. A probe that read provenance only up front reported it+absent, which is the opposite of the truth.++Nothing here raises. A payload that dies collecting a diagnostic reports nothing+at all, which is the one outcome worse than a missing field.+"""++from __future__ import annotations++# The kernels worth asking about for a Qwen3.5-class model, and what each answer+# is allowed to mean. `_EXPECTED` is deliberately NOT "all of these must be+# present": two of them are measured absent on this path and asserting them+# would be red on correct behaviour. See vision_kernel_failures below.+_KERNELS = ("fla", "causal_conv1d", "mamba_ssm", "flash_attn", "triton", "xformers")+++def probe_kernels() -> dict:+ """Import each kernel and record where it resolved from."""+ out: dict = {}+ for name in _KERNELS:+ entry: dict = {"importable": False}+ try:+ module = __import__(name)+ entry["importable"] = True+ entry["file"] = getattr(module, "__file__", None)+ entry["version"] = getattr(module, "__version__", None)+ # The distinction the whole module exists for. A vendored copy lives+ # inside unsloth_zoo; a pip-installed one does not.+ entry["vendored"] = "_vendored" in (entry["file"] or "")+ except BaseException as exc: # noqa: BLE001+ entry["error"] = f"{type(exc).__name__}: {exc}"[:200]+ out[name] = entry++ # Distributions separately, because a package can be installed and NOT+ # importable -- a wheel with the wrong CUDA ABI is exactly that -- and+ # reporting only the import would call that "absent".+ try:+ from importlib import metadata++ dists = {}+ for dist in metadata.distributions():+ name = (dist.metadata["Name"] or "").lower()+ if name in (+ "causal-conv1d",+ "mamba-ssm",+ "flash-attn",+ "fla-core",+ "flash-linear-attention",+ "xformers",+ ):+ dists[name] = dist.version+ out["_distributions"] = dists+ except Exception as exc: # noqa: BLE001+ out["_distributions"] = {"error": str(exc)[:200]}+ return out+++def attention_choice(model) -> dict:+ """What attention resolved to, read off the config.++ The config records the choice; a module walk was tried on the recon probe+ and returned an empty set, so this reports the one source that answers.+ """+ record: dict = {}+ try:+ config = getattr(model, "config", None)+ record["config"] = getattr(config, "_attn_implementation", None)+ text = getattr(config, "text_config", None)+ if text is not None:+ record["text_config"] = getattr(text, "_attn_implementation", None)+ except BaseException as exc: # noqa: BLE001+ record["error"] = f"{type(exc).__name__}: {exc}"[:200]+ return record+++def _is_turing(capability) -> bool:+ """True for compute capability 7.x, whichever way it was spelled."""+ text = str(capability or "").strip().lower().replace("sm_", "").replace("sm", "")+ if not text:+ return False+ # "7.5" -> 7, "75" -> 7. Both spellings appear in this repo.+ head = text.split(".")[0]+ if "." in text:+ return head == "7"+ return len(head) >= 2 and head[0] == "7"+++def vision_kernel_failures(+ kernels: dict | None, attention: dict | None, *, capability: str+) -> list:+ """The pass rule, as a pure function so it is checkable without a GPU.++ Three claims, each chosen so it is neither false nor vacuous on a T4:++ 1. **FLA is present and VENDORED.** Measured: it resolves to+ `unsloth_zoo/_vendored/fla`, version 0.5.1, after the load. Asserting+ merely "importable" would pass on a pip-installed copy that is not what+ ships, which is a different thing being tested.+ 2. **Attention is a valid Turing choice, and it is NOT flash_attention_2.**+ FA2 supports Ampere, Ada and Hopper. On sm_75 it cannot execute, so+ asserting it ran would be false and asserting it was "selected" would be+ vacuous. Asserting `sdpa` (or another real Turing path) is the claim+ that can be both true and informative -- and it catches the regression+ that matters, which is unsloth choosing a backend the card cannot run.+ 3. **Nothing was built from source.** A source build is silent and costs+ many minutes on 4 vCPUs.++ `causal_conv1d` and `mamba_ssm` are deliberately NOT asserted present.+ Measured on the recon probe: neither is installed on the notebook path,+ before or after the load. The wheel-first machinery in+ `studio/backend/utils/ssm_runtime.py` belongs to Studio's training worker+ and this path never calls it. Asserting them would be red on correct+ behaviour; they are REPORTED so a change shows up in the diff.+ """+ if not kernels:+ return ["no kernel provenance was collected at all"]++ failures = []++ fla = kernels.get("fla") or {}+ if not fla.get("importable"):+ failures.append(+ f"fla did not import after the model load, so the vendored fast "+ f"kernels are not reachable: {fla.get('error')}"+ )+ elif not fla.get("vendored"):+ failures.append(+ f"fla imported from {fla.get('file')!r}, which is not the vendored "+ f"copy under unsloth_zoo/_vendored. This leg is about the vendored "+ f"kernels; a pip-installed fla is a different thing"+ )++ # Normalised, because the two spellings in this repo are BOTH live and a+ # rule that silently never fires is the exact failure this file is about:+ # environment_fingerprint() records "sm_75", while the recon probe and+ # torch.cuda.get_device_capability report "7.5". A startswith("7.") check+ # against "sm_75" matches nothing, and an FA2 regression would sail past it.+ turing = _is_turing(capability)+ if turing:+ chosen = (attention or {}).get("config")+ if chosen in (None, ""):+ failures.append("no attention implementation was recorded")+ elif "flash_attention_2" in str(chosen):+ failures.append(+ f"attention resolved to {chosen!r} on capability {capability}. "+ f"FlashAttention-2 supports Ampere, Ada and Hopper; a Turing "+ f"card cannot run it, so this would fail at the first forward"+ )++ return failuresdiff --git a/tests/kaggle/t4_smoke/measured_vram.json b/tests/kaggle/t4_smoke/measured_vram.jsonnew file mode 100644index 00000000000..4257d6f6590--- /dev/null+++ b/tests/kaggle/t4_smoke/measured_vram.json@@ -0,0 +1,12 @@+{+ "_source": "executed payload notebooks of run 32611343797 (unslothai/unsloth), Tesla T4; multi_gpu from kernel unsloth-probe-multigpu-r2-a280e2, both cycles 1.2 GB",+ "_why": "Leg.vram_gb decides which legs may share a card and is checked against these. Recapture from a run's evidence, never by hand.",+ "card_total_gb": 14.56,+ "peak_reserved_gb": {+ "canary": 0.7,+ "control": 0.7,+ "frontier": 0.7,+ "gptoss": 12.78,+ "multi_gpu": 1.2+ }+}diff --git a/tests/kaggle/t4_smoke/naive_trl_compare.py b/tests/kaggle/t4_smoke/naive_trl_compare.pynew file mode 100644index 00000000000..62467525edb--- /dev/null+++ b/tests/kaggle/t4_smoke/naive_trl_compare.py@@ -0,0 +1,328 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""Train the same rows with PLAIN TRL, so an Unsloth trace has something to sit+next to.++**What this can say, and what it deliberately does not.** It does NOT assert+that the two paths produce the same losses, and the temptation to is the whole+reason this docstring is long. The `frontier` leg already showed transformers+5.5.0 and 5.15.1 producing different step-1 losses on identical weights, data+and seed (10.3222 against 6.4367), so the loss function itself moves between+library versions. Two DIFFERENT stacks agreeing to a tolerance would be a+coincidence, and a check built on one would go red on ordinary drift, which is+the kind of red that gets a check switched off the week before it is right.++What it asserts is the pair of claims that are version-independent: the plain+path **runs at all** on this model and this trl, and it **converges** rather+than sitting flat or going non-finite. Everything else is reported side by side+for a human.++**Why a separate file and a separate process.** Unsloth patches transformers,+trl and peft at import time. Anything that has imported it is no longer a+control, and an in-process "comparison" would be Unsloth against itself with+extra steps. This module must never import unsloth, and the guard in+tests/kaggle/test_t4_smoke_harness.py asserts that from the source rather than+trusting the convention.+"""++from __future__ import annotations++import argparse+import json+import os+import random+import sys+import time+from pathlib import Path++_HERE = Path(__file__).resolve().parent+++def _log(msg: str) -> None:+ print(f"[naive-trl] {msg}", flush = True)+++def _seed_everything(seed: int) -> None:+ import numpy as np+ import torch++ random.seed(seed)+ np.random.seed(seed)+ torch.manual_seed(seed)+ torch.cuda.manual_seed_all(seed)+++def _rows(path: Path) -> list[dict]:+ out = []+ for line in path.read_text(encoding = "utf-8").splitlines():+ line = line.strip()+ if line:+ out.append(json.loads(line))+ return out+++def run(args) -> dict:+ """Load in 4bit, attach LoRA, train, and report the step trace."""+ # Asserted rather than assumed: an unsloth already in sys.modules means+ # transformers is patched and this arm is not a control. It cannot happen+ # by accident in a fresh process, which is exactly why it would go+ # unnoticed if it ever did.+ if "unsloth" in sys.modules or "unsloth_zoo" in sys.modules:+ raise RuntimeError(+ "unsloth is imported in the plain-TRL process, so this is not a "+ "control arm; the comparison would be unsloth against itself"+ )++ import torch+ from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig++ result: dict = {+ "model": args.model,+ "max_steps": args.max_steps,+ "imported_unsloth": False,+ }++ _seed_everything(args.seed)++ # float16 and NF4, to match what the Unsloth arm ends up on: a T4 is sm_75+ # and has no bf16, and load_in_4bit there resolves to NF4 with a float16+ # compute dtype. Matching the numeric path is not enough to make the losses+ # comparable (see the module docstring) but mismatching it would add a+ # difference nobody asked about.+ quant = BitsAndBytesConfig(+ load_in_4bit = True,+ bnb_4bit_quant_type = "nf4",+ bnb_4bit_compute_dtype = torch.float16,+ bnb_4bit_use_double_quant = True,+ )++ t0 = time.time()+ tokenizer = AutoTokenizer.from_pretrained(args.model)+ model = AutoModelForCausalLM.from_pretrained(+ args.model,+ quantization_config = quant,+ dtype = torch.float16,+ device_map = {"": 0},+ )+ result["load_seconds"] = round(time.time() - t0, 1)+ result["resolved_checkpoint"] = getattr(getattr(model, "config", None), "_name_or_path", None)++ # The plain arm resolves its own repo, and it is NOT the one the Unsloth arm+ # loads: Unsloth's FLOAT_TO_INT_MAPPER redirects a 16bit name to a+ # pre-quantised `-unsloth-bnb-4bit` sibling, while this path quantises the+ # original on the fly. Recorded rather than reconciled, because forcing them+ # onto one repo would test a path neither a user nor the leg takes.++ from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training++ # Gradient checkpointing ON, and leaving it off was an unfair comparison+ # rather than a neutral one: the unsloth arm runs with+ # `gradient_checkpointing="unsloth"`, so a control without it is measured+ # with the single largest memory lever disabled on one side only. On+ # gemma-4-E2B-it that is the difference between a comparison and an OOM --+ # the control asked for 8.75GiB on top of 8.96GiB already resident, on a+ # 14.56GiB card (kernel unsloth-probe-latestcompile-r4-e67ef2).+ model = prepare_model_for_kbit_training(model, use_gradient_checkpointing = True)+ model = get_peft_model(+ model,+ LoraConfig(+ r = args.lora_r,+ lora_alpha = args.lora_alpha,+ lora_dropout = 0.0,+ bias = "none",+ task_type = "CAUSAL_LM",+ target_modules = [+ "q_proj",+ "k_proj",+ "v_proj",+ "o_proj",+ "gate_proj",+ "up_proj",+ "down_proj",+ ],+ ),+ )++ from datasets import Dataset+ from trl import SFTConfig, SFTTrainer++ rows = _rows(Path(args.dataset))+ eos = tokenizer.eos_token or ""+ dataset = Dataset.from_list([{"text": r["prompt"] + r["completion"] + eos} for r in rows])++ outdir = Path(args.outdir)+ outdir.mkdir(parents = True, exist_ok = True)++ config = SFTConfig(+ output_dir = str(outdir / "trainer"),+ per_device_train_batch_size = args.batch_size,+ gradient_accumulation_steps = args.grad_accum,+ max_steps = args.max_steps,+ learning_rate = args.learning_rate,+ logging_steps = 1,+ optim = args.optim,+ fp16 = True,+ bf16 = False,+ seed = args.seed,+ report_to = [],+ save_strategy = "no",+ max_length = args.max_seq_length,+ gradient_checkpointing = True,+ # Required with gradient checkpointing on a PEFT model: without it the+ # inputs carry no grad and the backward finds nothing to do, which+ # surfaces as "element 0 of tensors does not require grad" rather than+ # as a configuration mistake.+ gradient_checkpointing_kwargs = {"use_reentrant": False},+ )+ trainer = SFTTrainer(model = model, train_dataset = dataset, args = config)++ t0 = time.time()+ trainer.train()+ result["train_seconds"] = round(time.time() - t0, 1)++ metrics = []+ for entry in trainer.state.log_history:+ if "loss" in entry:+ metrics.append(+ {+ "step": int(entry.get("step", len(metrics) + 1)),+ "loss": entry["loss"],+ "grad_norm": entry.get("grad_norm"),+ }+ )+ result["metrics"] = metrics+ return result+++# Substrings that identify an out-of-memory failure, whichever layer raised it.+# torch says "CUDA out of memory", accelerate and bitsandbytes wrap it, and the+# exception TYPE is not reliably OutOfMemoryError once it has been re-raised.+_OOM_MARKERS = ("out of memory", "outofmemoryerror", "cuda oom")+++def _is_oom(text: str) -> bool:+ lowered = (text or "").lower()+ return any(marker in lowered for marker in _OOM_MARKERS)+++def comparison_failures(+ naive: dict | None,+ unsloth_metrics: list[dict] | None,+ *,+ allow_oom: bool = False,+) -> list[str]:+ """The two claims this comparison is entitled to make.++ Kept as a pure function of two dicts so the rules are testable on CPU+ without a GPU or a model, and separated from `run` so a guard cannot pass by+ exercising the rule against a hand-written dict alone -- see the Default+ leg, where seven such guards all passed while the code that produces the+ dict raised NameError on hardware.+ """+ if naive is None:+ return ["the plain-TRL arm produced no report at all"]+ if naive.get("error"):+ if allow_oom and _is_oom(naive["error"]) and not naive.get("metrics"):+ # An OOM BEFORE a single step is a statement about the card, not+ # about either training stack, and it is measured: on+ # gemma-4-E2B-it the plain arm asks for 8.75GiB with 8.96GiB+ # already resident on a 14.56GiB T4, and it does so at LOAD --+ # `metrics` is absent, so no step ever ran. Gradient checkpointing+ # does not touch that, and enabling it changed nothing.+ #+ # The likely cause is worth naming rather than implying: E2B is a+ # MatFormer SUBMODEL of E4B and the checkpoint carries the larger+ # weights, so a loader that does not extract the submodel+ # materialises all of them.+ #+ # Narrow on purpose. An OOM DURING training is still a failure --+ # that is a finding about the run, not about the card -- and this+ # only ever applies when the caller opted in.+ return []+ return [f"the plain-TRL arm did not run: {naive['error']}"]++ failures = []+ metrics = naive.get("metrics") or []+ if not metrics:+ failures.append(+ "the plain-TRL arm reported no steps, so it loaded and then trained "+ "nothing; a comparison against an empty trace is not a comparison"+ )+ return failures++ losses = [m.get("loss") for m in metrics]+ if any(+ not isinstance(v, (int, float)) or v != v or v in (float("inf"), float("-inf"))+ for v in losses+ ):+ failures.append(f"the plain-TRL arm produced a non-finite loss: {losses}")+ return failures++ # Converged, stated as "the end is below the start" rather than as a rate.+ # A tiny run on a tiny dataset has no business asserting a slope, but a+ # trace that FINISHES no lower than it started did not learn, and that is+ # the same rule the Unsloth arm is held to.+ if losses[-1] >= losses[0]:+ failures.append(+ f"the plain-TRL arm did not converge: first loss {losses[0]}, last "+ f"{losses[-1]}. Both arms are held to the same rule; this one is "+ f"not about agreeing with unsloth"+ )++ # Deliberately NOT compared for equality. Two library stacks do not produce+ # one fp16 trajectory, and asserting they do would be red on drift. The+ # count is checked instead, because a plain arm that silently ran fewer+ # steps is reported beside a full unsloth trace as though they were the+ # same experiment.+ if unsloth_metrics:+ if len(metrics) != len(unsloth_metrics):+ failures.append(+ f"the two arms ran different numbers of steps: plain TRL "+ f"{len(metrics)}, unsloth {len(unsloth_metrics)}. The traces are "+ f"reported side by side and a reader would compare them"+ )+ return failures+++def main() -> int:+ ap = argparse.ArgumentParser()+ ap.add_argument("--model", required = True)+ ap.add_argument("--dataset", default = str(_HERE / "canary_dataset.jsonl"))+ ap.add_argument("--outdir", required = True)+ ap.add_argument("--max-steps", type = int, default = 10)+ ap.add_argument("--batch-size", type = int, default = 2)+ ap.add_argument("--grad-accum", type = int, default = 1)+ ap.add_argument("--max-seq-length", type = int, default = 512)+ ap.add_argument("--learning-rate", type = float, default = 1e-3)+ ap.add_argument("--lora-r", type = int, default = 16)+ ap.add_argument("--lora-alpha", type = int, default = 32)+ ap.add_argument("--optim", default = "adamw_8bit")+ ap.add_argument("--seed", type = int, default = 3407)+ args = ap.parse_args()++ outdir = Path(args.outdir)+ outdir.mkdir(parents = True, exist_ok = True)+ report_path = outdir / "naive_trl_report.json"++ # A crash here must not take the leg down: this arm is a COMPARISON, and a+ # leg whose unsloth run passed should not go red because the control could+ # not install. The failure is recorded, reported, and left for+ # comparison_failures to rule on.+ try:+ result = run(args)+ except BaseException as exc: # noqa: BLE001+ text = str(exc).strip()+ result = {+ "model": args.model,+ "error": f"{type(exc).__name__}: {text}" if text else type(exc).__name__,+ }+ _log(f"failed: {result['error']}")++ report_path.write_text(json.dumps(result, indent = 2), encoding = "utf-8")+ print("NAIVE_TRL_REPORT " + json.dumps(result), flush = True)+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/tests/kaggle/t4_smoke/run_gptoss_t4.py b/tests/kaggle/t4_smoke/run_gptoss_t4.pyindex 95710f3930e..f2cc86c0922 100644--- a/tests/kaggle/t4_smoke/run_gptoss_t4.py+++ b/tests/kaggle/t4_smoke/run_gptoss_t4.py@@ -23,10 +23,15 @@ not in the path at all.probe saw `UNSLOTH_FORCE_FLOAT32=1`, `fp16=False`, `bf16=False`, and`UNSLOTH_FORCE_CUSTOM_DTYPE` pinning `down_projs` and `mlp.router` to float32.That path exists for this card and nothing else in CI exercises it.-* **No offload.** 12.78 GB reserved of 14.56, every parameter on `cuda:0`, no- `hf_device_map`. About 1.8 GB of headroom, thin enough that placement is- counted on every run rather than assumed -- and asserted, since a run that- offloads passes everything else.+* **One deliberate offload, and nothing else.** 12.7 GB reserved of 14.56, no+ `hf_device_map`, every parameter on `cuda:0` EXCEPT `model.embed_tokens.weight`,+ which unsloth puts in RAM on purpose -- `Unsloth: Offloading embeddings to RAM+ to save 1.08 GB`, with forward hooks carrying ids down and vectors back up.+ That was read as a spill twice before anyone looked at the name; 579133440 is+ exactly 201088 x 2880, this checkpoint's vocab by its hidden size. Placement is+ counted on every run rather than assumed, and the embedding is excused only+ when its hook flag is set, because an embedding that reached the CPU without+ them is a real bug that looks identical in a device count.* **torch.compile engages**: 32 unique graphs, 779 calls captured, 2 graphbreaks, both `_warnings.warn`. A silent fall back to eager leaves every othernumber healthy while the leg's coverage goes unexercised, so this is asserted.@@ -72,6 +77,8 @@ import jsonimport osimport platformimport sys+import shutil+import tempfileimport timeimport tracebackfrom pathlib import Path@@ -152,15 +159,51 @@ def placement(model) -> dict:landed on the CPU (correct, slow) from one on meta (loaded nothing)."""counts: dict = {}+ off_gpu: list = []try:- for param in model.parameters():+ for name, param in model.named_parameters():key = str(param.device)counts[key] = counts.get(key, 0) + param.numel()+ if not key.startswith("cuda"):+ # NAMED, not just counted. `{'cpu': 579133440}` is a number+ # nobody can act on; `model.embed_tokens.weight` is a bug+ # report. The two readings cost the same walk, and the first+ # one already reached hardware twice before anyone could say+ # which tensor it was.+ off_gpu.append({"name": name, "numel": param.numel(), "device": key})except Exception as exc: # noqa: BLE001counts = {"error": f"{type(exc).__name__}: {exc}"}+ # Is the one tensor allowed off the card the one unsloth DELIBERATELY put+ # there? `offload_embedding` moves the input embedding to RAM and installs+ # a pre/post forward hook pair that carries the ids down and the vectors+ # back up (`unsloth/models/vision.py:_install_offload_embedding_hooks`).+ # The flag it sets is the difference between that optimisation and a+ # weight that landed on the CPU by accident, and the two are identical in a+ # device count.+ embed = {}+ try:+ module = model.get_input_embeddings()+ weight = getattr(module, "weight", None)+ for name, candidate in model.named_modules():+ if candidate is module:+ embed["module"] = name+ break+ embed["weight_name"] = f"{embed.get('module')}.weight" if "module" in embed else None+ embed["device"] = str(weight.device) if weight is not None else None+ embed["offload_hooks_installed"] = bool(+ getattr(module, "_unsloth_offload_hooks_installed", False)+ )+ except Exception as exc: # noqa: BLE001+ embed = {"error": f"{type(exc).__name__}: {exc}"}+device_map = getattr(model, "hf_device_map", None)return {+ "input_embedding": embed,"parameters_by_device": counts,+ # Largest first, capped: a genuinely offloaded model has thousands of+ # these and the report is read by a human.+ "off_gpu_parameters": sorted(off_gpu, key = lambda p: -p["numel"])[:20],+ "off_gpu_parameter_count": len(off_gpu),"hf_device_map_devices": (sorted({str(v) for v in device_map.values()}) if isinstance(device_map, dict) else None),@@ -208,6 +251,91 @@ def build_dataset(tokenizer, rows: list[dict]):return Dataset.from_dict({"text": texts})+def build_completion_dataset(tokenizer, rows: list[dict]):+ """The same rows as a PROMPT/COMPLETION pair, so the loss covers only the+ answer.++ Two columns rather than a collator: TRL treats a dataset with `prompt` and+ `completion` columns as prompt-completion and masks the prompt itself+ (`completion_only_loss` defaults to True for that shape). The older+ `DataCollatorForCompletionOnlyLM` route needs a response template string+ that has to match the chat template exactly, and a template change turns it+ silently into "mask nothing" -- which trains on everything and passes every+ assertion about losses.++ The prompt ends with the generation prompt, so the boundary is exactly where+ the model would start generating. That is what makes the mask meaningful+ rather than approximately right.+ """+ from datasets import Dataset++ prompts, completions = [], []+ for row in rows:+ prompts.append(+ tokenizer.apply_chat_template(+ [{"role": "user", "content": row["question"]}],+ tokenize = False,+ add_generation_prompt = True,+ )+ )+ completions.append(row["answer"])+ return Dataset.from_dict({"prompt": prompts, "completion": completions})+++def masking_evidence(trainer) -> dict:+ """Whether the prompt tokens are ACTUALLY masked out of the loss.++ This is the whole point of the feature and the one thing a loss curve+ cannot show: a run that masks nothing trains on prompt and answer alike,+ converges perfectly well, and reports numbers indistinguishable from a+ correct one. So the labels are read off a real collated batch.++ Never raises: a diagnostic that kills the leg is worse than a missing one.+ """+ record: dict = {}+ try:+ batch = next(iter(trainer.get_train_dataloader()))+ labels = batch["labels"]+ total = int(labels.numel())+ masked = int((labels == -100).sum())+ record["label_tokens"] = total+ record["masked_tokens"] = masked+ record["masked_fraction"] = round(masked / total, 4) if total else None+ record["columns"] = sorted(batch.keys())+ except BaseException as exc: # noqa: BLE001+ record["error"] = f"{type(exc).__name__}: {exc}"[:400]+ return record+++def masking_failures(record: dict | None, *, expected: bool) -> list[str]:+ """The rule, as a pure function, so it is checkable without a GPU."""+ if not expected:+ return []+ if not record:+ return ["completions-only training was requested but no masking evidence was collected"]+ if record.get("error"):+ return [f"could not read the collated labels: {record['error']}"]++ total = record.get("label_tokens") or 0+ masked = record.get("masked_tokens")+ if not total:+ return ["the collated batch carried no labels at all"]+ if not masked:+ # The failure this function exists for. Every loss-based assertion+ # passes in this state.+ return [+ "completions-only training was requested and NOTHING was masked "+ f"({masked} of {total} label tokens are -100), so the prompt is in "+ f"the loss and this leg is not testing what it says it is"+ ]+ if masked >= total:+ return [+ f"every one of the {total} label tokens is masked, so there is no "+ f"completion left to learn from"+ ]+ return []++def train_and_infer(args) -> dict:import torchfrom unsloth import FastLanguageModel@@ -270,13 +398,22 @@ def train_and_infer(args) -> dict:for line in Path(args.dataset).read_text(encoding = "utf-8").splitlines()if line.strip()]- dataset = build_dataset(tokenizer, rows)+ # Prompt/completion when the loss should cover only the answer, one `text`+ # column otherwise. `dataset_text_field` must go with the second shape and+ # NOT the first: naming a text field TRL cannot find is how a+ # prompt-completion dataset silently falls back to training on everything.+ if args.train_on_completions:+ dataset = build_completion_dataset(tokenizer, rows)+ else:+ dataset = build_dataset(tokenizer, rows)+ result["train_on_completions"] = bool(args.train_on_completions)+ result["dataset_columns"] = sorted(dataset.column_names)from trl import SFTConfig, SFTTrainerconfig = SFTConfig(output_dir = str(Path(args.outdir) / "trainer"),- dataset_text_field = "text",+ **({} if args.train_on_completions else {"dataset_text_field": "text"}),per_device_train_batch_size = 1,gradient_accumulation_steps = 1,max_length = args.max_seq_length,@@ -309,6 +446,13 @@ def train_and_infer(args) -> dict:args = config,)+ # Read BEFORE training, off a real collated batch. After training the+ # dataloader has been consumed and a re-created one is not necessarily the+ # object the trainer used.+ if args.train_on_completions:+ result["masking"] = masking_evidence(trainer)+ _log(f"masking {json.dumps(result['masking'])}")+# The precision after Unsloth's patches have had their say. On a T4 this# should be float32 and NOT fp16; fp16 here means FORCE_FLOAT32 stopped# firing and the infinities it prevents are back.@@ -392,6 +536,87 @@ def train_and_infer(args) -> dict:result["generated"] = generatedresult["canary_found"] = CANARY in generatedresult["memory_peak"] = memory()++ # GGUF, LAST, because it merges a 20B checkpoint and the merge is the+ # heaviest thing in the leg. Anything after it would be measuring a session+ # that has just written ~28GB.+ #+ # ASK for q8_0, ACCEPT only mxfp4. That pairing looks backwards and is the+ # only one that works, which cost a probe to learn.+ #+ # gpt-oss overrides the request, out loud:+ #+ # GPT-OSS does not support GGUF quantization (requested: q8_0).+ # Overriding to MXFP4 format.+ # GPT-OSS model - skipping additional quantizations+ #+ # The obvious response is to ask for mxfp4 directly. unsloth REJECTS it as+ # an input -- measured on kernel unsloth-probe-gptoss-r3-832c85:+ #+ # Unsloth: Quant method = [mxfp4] not supported. Choose from below:+ # [not_quantized] [fast_quantized] [quantized] [f32] [bf16] ...+ #+ # So the documented override is the ONLY route to an MXFP4 file, and the+ # leg has to travel it. Accepting only mxfp4 is what keeps that honest: a+ # run that somehow produced a real q8_0 would fail, which is right, because+ # gpt-oss q8_0 is documented impossible.+ if getattr(args, "export_gguf", False):+ from gguf_export import export_gguf, llama_cpp_facts, run_gguf++ install_log = ""+ llama_dir = None+ try:+ import contextlib+ import io++ # After `import unsloth`, which has happened by now: unsloth_zoo's+ # llama_cpp raises "Please install Unsloth via pip install unsloth!"+ # if it is reached first. A probe already lost a session to that.+ from unsloth_zoo.llama_cpp import install_llama_cpp++ buffer = io.StringIO()+ with contextlib.redirect_stdout(buffer):+ returned = install_llama_cpp()+ install_log = buffer.getvalue()+ facts = llama_cpp_facts(install_log, returned)+ llama_dir = facts.get("dir")+ except BaseException as exc: # noqa: BLE001+ facts = {"error": f"{type(exc).__name__}: {exc}"[:2000]}+ _log(f"llama.cpp: {json.dumps(facts)}")++ # NOT under args.outdir, and this is measured. `/kaggle/working` is+ # 21.0GB total; the gpt-oss export consumes 27.6GB of transient disk+ # (3 mxfp4 safetensors shards at 13.76GB plus the GGUF). Exporting+ # there fails in 2.8s with+ # Unsloth: Failed saving locally - no disk space left+ # which is a disk fact wearing the costume of an export bug.+ #+ # `/tmp` on a Kaggle session is the overlay: 8656.9GB total, 1102.5GB+ # free, measured on kernel unsloth-probe-disk. tempfile honours TMPDIR+ # and lands there. The artifact keeps the RECORD -- path, size, seconds+ # -- and not the 27.6GB, which nobody wants collected anyway.+ gguf_dir = tempfile.mkdtemp(prefix = "gptoss_gguf_")+ record = export_gguf(+ model,+ tokenizer,+ gguf_dir,+ quantization = args.gguf_quantization,+ )+ record["disk_free_gb_before"] = round(shutil.disk_usage(gguf_dir).free / 1e9, 1)+ record["llama_cpp"] = facts+ result["gguf_export"] = record+ _log(f"gguf export: {json.dumps({k: v for k, v in record.items() if k != 'llama_cpp'})}")++ # Deliberately NOT run through llama.cpp here, and the reason is+ # measured: the prebuilt bundle is the -cpu build, and a 20B MXFP4 file+ # on 4 vCPUs is not a few seconds of work. The Default leg already+ # covers "the exported file runs" on a 610MB Q8_0, where the claim is+ # affordable. Asserting the export and not the run is a smaller claim,+ # and it is the one this leg can honestly make.+ ggufs = record.get("ggufs") or []+ result["gguf_ran"] = False+ if ggufs:+ _log(f"largest gguf: {ggufs[0]['path']} ({ggufs[0]['mb']} MB), not executed here")_log(f"generated {generated!r}")return result@@ -427,13 +652,41 @@ def _placement_failures(placement: dict | None) -> list[str]:device: n for device, n in counts.items() if not str(device).startswith("cuda")}if elsewhere:- failures.append(- f"parameters are off the GPU: {elsewhere} (all devices: {counts}). "- f"This leg's result is that the 20B checkpoint fits and trains "- f"wholly on one T4; a run that offloads to CPU, disk or meta is "- f"not a slower version of that, it is a different run, and "- f"accelerate's offload does not support training at all"+ # The ONE tensor allowed off the card, and only on its own terms.+ # `offload_embedding` puts the input embedding in RAM and hooks the+ # lookup so ids go down and vectors come back up; measured saving+ # 1.08GB on gpt-oss-20b. Accepting "cpu" wholesale would excuse a+ # real spill, so this names the exact parameter AND requires the+ # hook flag: an embedding that reached the CPU without them is a+ # bug, and it looks identical in a device count.+ embed = placement.get("input_embedding") or {}+ deliberate = (+ embed.get("offload_hooks_installed") is True+ and str(embed.get("device", "")).startswith("cpu")+ and embed.get("weight_name"))+ unexplained = [+ p+ for p in (placement.get("off_gpu_parameters") or [])+ if not (deliberate and p.get("name") == embed.get("weight_name"))+ ]+ if unexplained or not (placement.get("off_gpu_parameters") or []):+ named = (+ ", ".join(+ f"{p.get('name')} ({p.get('numel')} on {p.get('device')})"+ for p in unexplained+ )+ or "the walk recorded no names"+ )+ failures.append(+ f"parameters are off the GPU: {elsewhere} [{named}] (all devices: "+ f"{counts}), and unsloth's deliberate embedding offload does not "+ f"account for them (input_embedding {embed}). This leg's result is "+ f"that the 20B checkpoint fits and trains wholly on one T4; a run "+ f"that offloads to CPU, disk or meta is not a slower version of "+ f"that, it is a different run, and accelerate's offload does not "+ f"support training at all"+ )# The accelerate-side answer, which is absent on a healthy run and says# `cpu`/`disk` when dispatch offloaded. Three-way: `placement()` always# writes a bool, so anything else is a record this file cannot read.@@ -455,6 +708,16 @@ def failures_for(result: dict, args) -> list[str]:that costs a Kaggle session has to be checkable without one."""failures: list[str] = []+ if getattr(args, "export_gguf", False):+ from gguf_export import export_failures++ # MXFP4 only. gpt-oss refuses every other quantization by design, so a+ # wider accept list would let a silently-overridden export pass as+ # though the request had been honoured.+ failures += export_failures(result.get("gguf_export"), accept_quantizations = ("mxfp4",))+ failures += masking_failures(+ result.get("masking"), expected = bool(getattr(args, "train_on_completions", False))+ )metrics = result.get("metrics") or []if len(metrics) != args.max_steps:failures.append(f"expected {args.max_steps} logged steps, got " f"{len(metrics)}")@@ -619,6 +882,20 @@ def main() -> int:ap.add_argument("--max-seq-length", type = int, default = 1024)ap.add_argument("--lora-r", type = int, default = 8)ap.add_argument("--max-new-tokens", type = int, default = 32)+ ap.add_argument(+ # On by default: the leg is specified to train on completions, and a+ # flag that has to be remembered at every call site is one that will be+ # forgotten at one of them.+ "--train-on-completions",+ dest = "train_on_completions",+ action = "store_true",+ default = True,+ )+ ap.add_argument("--no-train-on-completions", dest = "train_on_completions", action = "store_false")+ ap.add_argument("--export-gguf", dest = "export_gguf", action = "store_true", default = False)+ # q8_0 as the REQUEST. `mxfp4` is not an accepted input value; the override+ # is the only way to reach that format. See the export block above.+ ap.add_argument("--gguf-quantization", default = "q8_0")ap.add_argument("--require-compile", dest = "require_compile", action = "store_true", default = True)ap.add_argument("--no-require-compile", dest = "require_compile", action = "store_false")ap.add_argument(diff --git a/tests/kaggle/t4_smoke/run_grpo_t4.py b/tests/kaggle/t4_smoke/run_grpo_t4.pyindex c4242ace2b9..c0b9bef3c28 100644--- a/tests/kaggle/t4_smoke/run_grpo_t4.py+++ b/tests/kaggle/t4_smoke/run_grpo_t4.py@@ -114,16 +114,38 @@ def _texts(completions) -> list[str]:return out+# Chosen so the reward can never saturate. len/(len+SCALE) is strictly+# increasing on [0, inf), maps into (0, 1), and equals 0.5 at SCALE characters,+# so it stays informative at every length instead of only below a cap.+_LENGTH_SCALE = 200.0++def reward_length(completions, **kwargs) -> list[float]:- """Longer completions score higher, saturating at 200 characters.+ """Longer completions score higher, without ever saturating.Deterministic given the text, yet SENSITIVE to a group's diversity: a- constant reward would zero `reward_std` on a healthy run and destroy the- only instrument this leg has.+ constant reward zeroes `reward_std` on a healthy run and destroys the only+ instrument this leg has.++ THAT IS NOT HYPOTHETICAL - this function used to be++ min(len(t), 200) / 200.0++ and the docstring above already claimed it was diversity-sensitive. It was+ not. Kernels unsloth-probe-grpo-rep2-b03be8 and -rep3-bc3828 recorded+ completions of 2534 to 3396 characters, an order of magnitude past the cap,+ so every completion scored exactly 1.0, every group tied, and the leg failed+ with `reward_std was zero on every step` - a red that looks like a broken+ generation path and is really a reward that stopped measuring anything.+ Two of three runs died this way, which made it the leg's dominant failure.++ The cap was the whole problem: it was set BELOW the lengths the model+ actually produces, so the one region it discriminated in was the one region+ the run never visited."""texts = _texts(completions)SEEN_COMPLETIONS.append(texts)- return [min(len(t), 200) / 200.0 for t in texts]+ return [len(t) / (len(t) + _LENGTH_SCALE) for t in texts]def reward_digit(completions, **kwargs) -> list[float]:diff --git a/tests/kaggle/t4_smoke/run_t4_smoke.py b/tests/kaggle/t4_smoke/run_t4_smoke.pyindex 6908c798673..d8ffd77b8c9 100644--- a/tests/kaggle/t4_smoke/run_t4_smoke.py+++ b/tests/kaggle/t4_smoke/run_t4_smoke.py@@ -67,6 +67,7 @@ import jsonimport osimport platformimport subprocess+import tempfileimport sysimport timefrom pathlib import Path@@ -84,6 +85,8 @@ from determinism import ( # noqa: E402set_all_seeds_fast,set_deterministic_algorithms,)+from kernel_provenance import attention_choice, probe_kernels, vision_kernel_failures # noqa: E402+from naive_trl_compare import comparison_failures # noqa: E402from training_evidence import LORA_B_MARKER, LORA_MARKER # noqa: E402from versions import ( # noqa: E402GOAL_PACKAGES,@@ -266,11 +269,40 @@ def train_once(args, run_index: int) -> dict:# float16 unconditionally: T4 is sm_75 and has no bf16. Pinned rather than# left to the loader so the local reproduction and the Kaggle run take the# same numeric path wherever they can share one.+ load_kwargs: dict = {}+ if getattr(args, "single_device", False):+ # Both cards stay VISIBLE -- that is the point of the multi_gpu leg, and+ # what makes unsloth's DEVICE_COUNT > 1 bindings live -- but the weights+ # go on one, because a SHARDED model does not train.+ #+ # Measured on unsloth-probe-multigpu-r1-18beab: with two T4s visible+ # accelerate split Qwen3-0.6B 232849408 params on cuda:0 and 155582464+ # on cuda:1, unsloth printed `Num GPUs used = 2`, and step 0 died in+ # unsloth/models/llama.py:972 --+ #+ # inputs_embeds = self.embed_tokens(input_ids)+ # RuntimeError: Expected all tensors to be on the same device, but got+ # index is on cuda:0, different from other tensors on cuda:1+ #+ # WHETHER THAT IS A BUG IS NOT SETTLED, and an earlier version of this+ # comment claimed it was filed. It is not filed, and the repo's own+ # history is split on it: #2467 was closed with "unsloth runs on only 1+ # gpu.. It wont run in multi gpu systems. force it to use only 1 gpu",+ # while #2882 -- the same message, a different call site -- was closed+ # as fixed. So this may be documented behaviour rather than a defect.+ #+ # What is not in doubt is that it is not THIS repo's to work around in a+ # per-PR check. Training a sharded model here would put a red in front+ # of every PR for a fault no reader can act on, which is how a check+ # gets switched off before the day it is right. The binding coverage --+ # the whole reason this leg exists -- does not depend on it.+ load_kwargs["device_map"] = {"": 0}model, tokenizer = FastLanguageModel.from_pretrained(model_name = args.model,max_seq_length = args.max_seq_length,load_in_4bit = True,dtype = torch.float16,+ **load_kwargs,)load_seconds = time.time() - t0# Which repository, and which commit of it. `load_in_4bit=True` redirects@@ -284,6 +316,26 @@ def train_once(args, run_index: int) -> dict:resolved_revision = getattr(_config, "_commit_hash", None)_log(f"loaded {resolved_checkpoint} @ {resolved_revision}")+ # AFTER the load and not before, which is measured rather than tidy: on+ # kernel unsloth-probe-vision-recon-c76ea3 `fla` was NOT importable before+ # from_pretrained and WAS after, because unsloth reaches for it lazily when+ # it sees the model. Reading provenance up front reports it absent.+ if getattr(args, "kernel_provenance", False):+ result_kernels = probe_kernels()+ result_attention = attention_choice(model)+ _log(f"kernels: {json.dumps(result_kernels)}")+ _log(f"attention: {json.dumps(result_attention)}")+ else:+ result_kernels = None+ result_attention = None++ # AFTER the load for the same reason as the block above: the rotary caches+ # are built when the model is, so reading them before from_pretrained finds+ # nothing and reports it as an absence.+ result_multi_gpu = multi_gpu_facts(model) if getattr(args, "require_multi_gpu", False) else None+ if result_multi_gpu is not None:+ _log(f"multi-gpu: {json.dumps(result_multi_gpu)}")+# Bound to a name so the saved config is checked against the adapter that# was actually requested, rather than against a list repeated further down.target_modules = [@@ -401,7 +453,16 @@ def train_once(args, run_index: int) -> dict:# function of the weights alone.FastLanguageModel.for_inference(model)prompt = PROMPT_TEMPLATE.format(question = rows[0]["question"])- inputs = tokenizer([prompt], return_tensors = "pt").to(model.device)+ # `text = ` and not positional, and this cost a leg. A vision model's+ # tokenizer IS a processor, whose signature is+ # `__call__(self, images=None, text=None, videos=None, ...)` -- so a+ # positional list is taken as IMAGES and transformers tries to fetch the+ # prompt strings as image URLs. Measured on kernel+ # unsloth-probe-latestcompile-r2-62b54d with gemma-4-E2B-it:+ # transformers/image_processing_backends.py, in fetch_images+ # `text` is the first parameter of a plain tokenizer too, so the keyword is+ # correct for both and is not a special case for vision.+ inputs = tokenizer(text = [prompt], return_tensors = "pt").to(model.device)t0 = time.time()with torch.inference_mode():out = model.generate(@@ -417,16 +478,123 @@ def train_once(args, run_index: int) -> dict:infer_seconds = time.time() - t0generated = tokenizer.decode(out[0][inputs["input_ids"].shape[1] :], skip_special_tokens = True)+ # Batched generation, on the SAME trained model, immediately after the+ # single-prompt generation above. Prompts of deliberately different lengths,+ # because a batch of equal-length prompts pads nothing and would report a+ # green left-padding check that never padded.+ batch_prompts = [+ PROMPT_TEMPLATE.format(question = row["question"]) for row in rows[: max(BATCH_SIZES)]+ ]+ while len(batch_prompts) < max(BATCH_SIZES):+ # The canary dataset is small. Pad the LIST (not the tensors) by+ # reusing questions with a varying prefix, which keeps the token+ # lengths spread rather than repeating one length.+ idx = len(batch_prompts)+ batch_prompts.append(+ PROMPT_TEMPLATE.format(+ question = " ".join(["please"] * (idx % 5 + 1))+ + " "+ + rows[idx % len(rows)]["question"]+ )+ )+ batched = batched_generation(+ model,+ tokenizer,+ batch_prompts,+ max_new_tokens = args.max_new_tokens,+ )+ _log(f"batched generation: {json.dumps({k: v for k, v in batched.items() if k != 'batched'})}")++ # GGUF export, opt-in. Placed AFTER generation deliberately: the export+ # merges the adapter into the base weights, and doing that before the+ # canary and batched-generation checks would have them measure a different+ # model from the one training produced.+ gguf_export_record = None+ gguf_run_record = None+ # ONCE per leg, on the first cycle, not once per cycle. Measured: the+ # conversion is 310.8s and 312.3s on the two Latest_compile cycles and+ # 99.3s and 117.3s on the two vision ones, so the repeat is 47% of the+ # longest leg in the suite. The second export is the same base weights and+ # an adapter trained by the same script with the same seed, so it re-runs+ # llama.cpp rather than asking a new question; reproducibility is already+ # asserted on the step tables and on the generated text.+ if getattr(args, "export_gguf", False) and run_index > 0:+ gguf_export_record = {+ "skipped": "exported on cycle 0; the conversion is the same weights twice"+ }+ elif getattr(args, "export_gguf", False):+ from gguf_export import export_gguf, llama_cpp_facts, run_gguf++ # unsloth must be imported before unsloth_zoo.llama_cpp, which raises+ # "Please install Unsloth via pip install unsloth!" otherwise. It is,+ # by the time train_once runs, but the import stays local so a payload+ # that never exports does not pay for it.+ install_log = ""+ llama_dir = None+ try:+ import contextlib+ import io++ from unsloth_zoo.llama_cpp import install_llama_cpp++ buffer = io.StringIO()+ with contextlib.redirect_stdout(buffer):+ returned = install_llama_cpp()+ install_log = buffer.getvalue()+ facts = llama_cpp_facts(install_log, returned)+ llama_dir = facts.get("dir")+ except BaseException as exc: # noqa: BLE001+ facts = {"error": f"{type(exc).__name__}: {exc}"[:2000]}+ _log(f"llama.cpp: {json.dumps(facts)}")++ # NOT under args.outdir. That is `/kaggle/working`, which is 21.0 GB+ # and is also what `kernels output` ships back, so a merge that fits+ # there is downloaded and a merge that does not kills the leg. Measured+ # on unsloth-probe-lcleg-final-a90fbb:+ #+ # RuntimeError: Unsloth: Not enough disk space to convert to GGUF.+ # The export needs about 16.6GB on the filesystem holding+ # `/kaggle/working/t4_out_Latest_compile/cycle0/gguf_run0`+ #+ # /tmp is the same overlay as `/`, 8656.9 GB with ~1100 GB free, which+ # this file has recorded since the first disk probe. tempfile honours+ # TMPDIR and lands there. The RECORD travels back -- path, size,+ # seconds -- and the multi-gigabyte artifact does not, which nobody+ # wanted collected anyway.+ #+ # gpt-oss and the vision run were both moved for this exact reason and+ # this path was missed, which is why a small model kept passing.+ gguf_export_record = export_gguf(+ model,+ tokenizer,+ tempfile.mkdtemp(prefix = f"gguf_run{run_index}_"),+ quantization = args.gguf_quantization,+ )+ gguf_export_record["llama_cpp"] = facts+ _log(+ f"gguf export: {json.dumps({k: v for k, v in gguf_export_record.items() if k != 'llama_cpp'})}"+ )++ ggufs = gguf_export_record.get("ggufs") or []+ if ggufs and llama_dir:+ gguf_run_record = run_gguf(ggufs[0]["path"], llama_dir)+peak_gb = torch.cuda.max_memory_reserved() / 1024**3 if torch.cuda.is_available() else 0.0result = {"run_index": run_index,+ "kernels": result_kernels,+ "attention": result_attention,+ "multi_gpu": result_multi_gpu,"metrics": stats.logs,"generated": generated,# Both, because they answer different questions when this goes red:# `canary_found` says training reached the weights at all, `canary_exact`# is the assertion, and the gap between them is the signature of a# stopping/EOS regression rather than a training one.+ "batched_generation": batched,+ "gguf_export": gguf_export_record,+ "gguf_run": gguf_run_record,"canary_found": CANARY in generated,"canary_exact": generated.strip() == CANARY,"prompt": prompt,@@ -511,6 +679,27 @@ def _reconstruct_adapter_config(adapter_dir, expected: dict | None) -> dict:unchecked.append(key)continuegot = getattr(config, key)+ if key == "target_modules" and isinstance(got, str):+ # PEFT allows target_modules to be a REGEX as well as a list, and+ # unsloth writes one for vision models so the adapter targets the+ # language tower and not the vision encoder. Measured on+ # unsloth-probe-vision-leg-r2-793ec0 with Qwen3.5-2B and again on+ # gemma-4-E2B-it, where the saved value begins+ # (?:.*?(?:language|text).*?(?:self_attn|attention|...+ #+ # Comparing that string against the list that was REQUESTED reports+ # a difference on every vision model, which is correct behaviour+ # being called a defect. The claim worth keeping is narrower and+ # still falsifiable: every module name asked for must appear in the+ # pattern, so a silently dropped projection is still caught.+ missing = [name for name in (wanted or []) if name not in got]+ same = not missing+ if not same:+ differences.append(+ f"{key}: trained with {wanted!r}, and the saved regex does "+ f"not mention {missing!r}: {got!r}"+ )+ continueif isinstance(wanted, (list, tuple, set)) or isinstance(got, (list, tuple, set)):same = sorted(got or []) == sorted(wanted or [])else:@@ -522,6 +711,307 @@ def _reconstruct_adapter_config(adapter_dir, expected: dict | None) -> dict:return out+# Batch sizes to cross-check against one-at-a-time generation. 1 is the+# baseline and is generated separately; the rest must reproduce it exactly.+BATCH_SIZES = (2, 4, 8)+++def batched_generation(model, tokenizer, prompts, *, max_new_tokens) -> dict:+ """Greedy generation one-at-a-time, then batched, and whether they agree.++ WHAT THIS IS FOR. Batched generation with left padding has broken here+ before, repeatedly and in ways that pass every other check in this file:++ * #3699 batched generation with left-padding and caching produced incorrect+ output,+ * #1066 batch inference produced gibberish,+ * #1456 batch inference was inconsistent for a self-trained model,+ * #2138 a release silently FORCED the tokenizer padding side to right during+ inference, which is why the side is recorded as OBSERVED after generating+ rather than as the value this function set.++ Greedy decoding makes the comparison meaningful: the output is then a+ function of the weights and the attention mask alone, so any difference+ between batch sizes is padding or cache handling rather than sampling.++ THE VACUITY TRAP, and it is the whole reason this returns the token lengths:+ padding only happens when the prompts in a batch have DIFFERENT lengths.+ A batch of equal-length prompts pads nothing, agrees trivially, and reports+ a green left-padding check that never once left-padded. The caller asserts+ the spread; this function measures it.+ """+ # Imported here, not at module scope, matching every other torch user in+ # this file: the module is loaded before unsloth is installed in some+ # paths, and a top-level torch import would move the failure to import+ # time. Kernel unsloth-probe-defaultleg-723c28 trained all 10 steps and+ # then died on `NameError: name 'torch' is not defined` right here.+ import torch++ tokenizer.padding_side = "left"+ if tokenizer.pad_token is None:+ tokenizer.pad_token = tokenizer.eos_token++ def _gen(batch: list) -> list:+ # Keyword for the same reason as the single-prompt path above: a+ # processor reads a positional list as images.+ enc = tokenizer(text = batch, return_tensors = "pt", padding = True).to(model.device)+ with torch.inference_mode():+ out = model.generate(+ **enc,+ max_new_tokens = max_new_tokens,+ do_sample = False,+ temperature = None,+ top_p = None,+ top_k = None,+ use_cache = True,+ # `x or y` is wrong here: pad_token_id 0 is a perfectly ordinary+ # id (Qwen and Llama both use low ids) and is falsy, so `or`+ # silently substitutes the EOS id for it and pads the batch with+ # end-of-sequence tokens. Test for None.+ pad_token_id = (+ tokenizer.pad_token_id+ if tokenizer.pad_token_id is not None+ else tokenizer.eos_token_id+ ),+ )+ # Slice by the PADDED width, not by each prompt's own length: with left+ # padding every row starts at the same column, and using the unpadded+ # length would re-read the tail of the prompt as if it were output.+ width = enc["input_ids"].shape[1]+ return [tokenizer.decode(row[width:], skip_special_tokens = True) for row in out]++ # `[0]`, and the missing index made this whole check vacuous. A processor+ # returns input_ids with a BATCH dimension, so `len(...)` on it is the+ # number of sequences -- 1 -- for every prompt. Measured on kernel+ # unsloth-probe-latestcompile-r3-cb1125, where gemma-4 reported+ # [1, 1, 1, 1, 1, 1, 1, 1] and the vacuity guard below caught it:+ # "every batched prompt tokenised to the same length ... so nothing was+ # ever padded and the left-padding check proved nothing"+ # A plain tokenizer given one string returns a flat list, which is why this+ # read correctly on every text model and only broke on the first vision one.+ lengths = [len(tokenizer(text = [p])["input_ids"][0]) for p in prompts]+ singles = [_gen([p])[0] for p in prompts]+ result = {+ "prompt_token_lengths": lengths,+ "distinct_lengths": len(set(lengths)),+ "padding_side_observed": tokenizer.padding_side,+ "singles": singles,+ "batched": {},+ "agrees": {},+ "empty_outputs": [i for i, text in enumerate(singles) if not text.strip()],+ }+ for size in BATCH_SIZES:+ outs: list = []+ for start in range(0, len(prompts), size):+ outs.extend(_gen(prompts[start : start + size]))+ result["batched"][str(size)] = outs+ result["agrees"][str(size)] = outs == singles+ # Read AGAIN, after all the generating. #2138 was a silent override applied+ # inside the inference path, so the value set at the top of this function is+ # not evidence of the value that was used.+ result["padding_side_after"] = tokenizer.padding_side+ return result+++# Models where batched greedy generation is CONFIRMED broken upstream, filed,+# and not this repo's to fix. See unsloth #9708: on both of these, batch sizes+# 2/4/8 fail to reproduce one-at-a-time output with real left padding and+# demonstrably distinct prompt lengths, on both repeats.+#+# This is a STRICT expectation, not a mute. A model listed here that starts+# AGREEING fails the leg, with a message saying to delete the entry -- so the+# day #9708 is fixed, CI says so instead of quietly keeping a stale excuse.+# Every other rule in this function stays live for these models: the padding+# side, the distinct lengths and the empty-output check are what make the+# disagreement a real finding rather than an unpadded batch.+KNOWN_BATCHED_GENERATION_BREAKAGE = {+ "unsloth/gemma-4-E2B-it": "unsloth#9708",+ "unsloth/Qwen3.5-2B": "unsloth#9708",+}+++def batched_generation_failures(batch: dict | None, model: str | None = None) -> list[str]:+ """Turn a `batched_generation` record into failures, vacuity included."""+ if not batch:+ return ["batched generation was never run"]+ out = []+ known = KNOWN_BATCHED_GENERATION_BREAKAGE.get(model or "")+ if batch.get("distinct_lengths", 0) < 2:+ out.append(+ "every batched prompt tokenised to the same length "+ f"({batch.get('prompt_token_lengths')}), so nothing was ever padded "+ "and the left-padding check proved nothing"+ )+ if len(batch.get("singles") or []) < max(BATCH_SIZES):+ out.append(+ f"only {len(batch.get('singles') or [])} prompts for a batch size of "+ f"{max(BATCH_SIZES)}, so the largest batch was never actually formed"+ )+ for side_key in ("padding_side_observed", "padding_side_after"):+ if batch.get(side_key) != "left":+ out.append(+ f"{side_key} is {batch.get(side_key)!r}, not 'left'; a right-padded "+ f"decoder-only batch attends to pad tokens before the prompt (#2138)"+ )+ if batch.get("empty_outputs"):+ out.append(f"prompts {batch['empty_outputs']} generated nothing at all")+ agrees = batch.get("agrees") or {}+ for size, agreed in agrees.items():+ if not agreed and not known:+ out.append(+ f"batch size {size} did not reproduce one-at-a-time greedy output "+ f"(#3699/#1456): {batch.get('batched', {}).get(size)!r} != "+ f"{batch.get('singles')!r}"+ )+ if known and agrees and all(agrees.values()):+ # The strict half. An expectation that only ever excuses is a mute, and+ # a mute outlives the bug it was written for.+ out.append(+ f"{model} is listed in KNOWN_BATCHED_GENERATION_BREAKAGE for "+ f"{known}, and every batch size AGREED. If the upstream fix has "+ f"landed, delete the entry; the leg must not carry an excuse for a "+ f"bug that is gone"+ )+ return out+++# Set by multi_gpu_facts as soon as it has an answer, so a crash LATER in the+# cycle still reports what was measured. A reading that exists only inside a+# frame that is being unwound is a reading nobody has.+_LAST_MULTI_GPU_FACTS: dict | None = None+++def multi_gpu_facts(model) -> dict:+ """What unsloth BOUND, given how many cards this process can see.++ The point of the multi_gpu leg is a branch no pinned payload can reach.+ `unsloth/kernels/utils.py:170`:++ if DEVICE_COUNT > 1:+ torch_gpu_device = torch.cuda.device # a real device switch+ else:+ def torch_gpu_device(device): return nullcontext()++ `build_kernel.py` pins every ordinary payload with CUDA_VISIBLE_DEVICES, so+ every unsloth kernel this CI has run took the nullcontext branch, and so did+ the DEVICE_COUNT-sized CUDA_STREAMS / WEIGHT_BUFFERS / ABSMAX_BUFFERS arrays+ and the per-device rotary caches in unsloth/models/llama.py.++ Read off the IMPORTED MODULE, not recomputed from `device_count()`. The+ binding is made once at import time, so asking torch how many cards there+ are answers a different question -- and answers it the way the check wants,+ which is the shape of a rule that cannot fail.+ """+ import torch++ global _LAST_MULTI_GPU_FACTS+ facts: dict = {"device_count": torch.cuda.device_count()}+ _LAST_MULTI_GPU_FACTS = facts+ try:+ from unsloth.kernels import utils as _kernel_utils+ except BaseException as exc: # noqa: BLE001+ facts["error"] = f"{type(exc).__name__}: {exc}"+ return facts++ binding = getattr(_kernel_utils, "torch_gpu_device", None)+ facts["module_device_count"] = getattr(_kernel_utils, "DEVICE_COUNT", None)+ # `torch.cuda.device` is a CLASS; the single-card fallback is a module-level+ # function closing over nullcontext. Identity against the real thing rather+ # than a name check, because both are bound to the same name.+ facts["torch_gpu_device_is_real_switch"] = binding is torch.cuda.device+ facts["torch_gpu_device_repr"] = repr(binding)[:200]+ for name in ("CUDA_STREAMS", "WEIGHT_BUFFERS", "ABSMAX_BUFFERS"):+ value = getattr(_kernel_utils, name, None)+ facts[name.lower() + "_len"] = None if value is None else len(value)++ # Where the weights actually sit. With two cards visible accelerate may+ # shard, and whether it does is a MEASUREMENT rather than something to+ # assert blind: kernel unsloth-probe-vision-recon-c76ea3 saw a model split+ # 897.7 MB / 1017.1 MB across two T4s, but that was a different model and a+ # different loader path.+ by_device: dict[str, int] = {}+ try:+ for _, param in model.named_parameters():+ key = str(param.device)+ by_device[key] = by_device.get(key, 0) + param.numel()+ except BaseException as exc: # noqa: BLE001+ facts["parameter_walk_error"] = f"{type(exc).__name__}: {exc}"+ facts["parameters_by_device"] = by_device+ facts["cuda_devices_holding_parameters"] = sorted(d for d in by_device if d.startswith("cuda"))++ # The per-device rotary caches, sized by DEVICE_COUNT at construction. A+ # list of length 1 on a two-card box means the model was built while+ # unsloth believed there was one card.+ for module in getattr(model, "modules", lambda: [])():+ cached = getattr(module, "multi_gpu_cos_cached", None)+ if cached is not None:+ facts["rotary_cache_slots"] = len(cached)+ break+ return facts+++def multi_gpu_failures(facts: dict | None, *, expected_cards: int) -> list[str]:+ """The rules, separated from the reading so they can be driven on CPU.++ Deliberately NOT asserting that the parameters are spread across both+ cards. Whether unsloth shards them or pins them to cuda:0 is exactly what+ this leg is being run to find out, and a rule written before the answer is+ a rule written to match whatever happens.+ """+ if not facts:+ return [+ "the multi-GPU facts are missing, so nothing about the "+ "DEVICE_COUNT > 1 path was measured on this run"+ ]+ if facts.get("error"):+ return [f"unsloth.kernels.utils could not be read: {facts['error']}"]++ out: list[str] = []+ seen = facts.get("device_count")+ if seen != expected_cards:+ out.append(+ f"this leg exists to exercise the multi-card path and torch sees "+ f"{seen} card(s), not {expected_cards} -- the driver pinned it, so "+ f"every assertion below would measure the single-card branch under "+ f"a multi-GPU name"+ )+ # Everything after this measures the wrong machine, so say so once.+ return out++ if facts.get("module_device_count") != expected_cards:+ out.append(+ f"unsloth.kernels.utils.DEVICE_COUNT is "+ f"{facts.get('module_device_count')!r} while torch sees "+ f"{expected_cards}; the module was imported before the cards were "+ f"visible, so its bindings are the single-card ones"+ )+ if not facts.get("torch_gpu_device_is_real_switch"):+ out.append(+ f"torch_gpu_device is not torch.cuda.device but "+ f"{facts.get('torch_gpu_device_repr')!r} -- the nullcontext "+ f"fallback, which performs NO device switch, so this run covered "+ f"the same path a pinned leg already covers"+ )+ for name in ("cuda_streams", "weight_buffers", "absmax_buffers"):+ length = facts.get(name + "_len")+ if length is None or length < expected_cards:+ out.append(+ f"unsloth.kernels.utils.{name.upper()} has {length!r} entries "+ f"for {expected_cards} cards, so a dequant on the second card "+ f"has no stream or buffer of its own"+ )+ slots = facts.get("rotary_cache_slots")+ if slots is not None and slots < expected_cards:+ out.append(+ f"the per-device rotary cache has {slots} slot(s) for "+ f"{expected_cards} cards, so the model was built while unsloth "+ f"believed there was one"+ )+ if not facts.get("cuda_devices_holding_parameters"):+ out.append("no parameter is on a CUDA device at all")+ return out++def peft_adapter_keys(model) -> dict:"""The names PEFT itself gives this adapter's tensors, off the live model.@@ -1278,6 +1768,38 @@ def optimisation_failures(metrics: list[dict]) -> list[str]:def main() -> int:ap = argparse.ArgumentParser()ap.add_argument("--model", default = DEFAULT_MODEL)+ # On by default: batched generation is the surface that has broken most+ # often here (#3699, #1066, #1456, #2138) and a leg that quietly skips it+ # is a leg that stops covering it.+ ap.add_argument(+ "--check-batched-generation",+ dest = "check_batched_generation",+ action = "store_true",+ default = True,+ )+ ap.add_argument(+ "--no-check-batched-generation",+ dest = "check_batched_generation",+ action = "store_false",+ )+ # OFF by default, unlike the batched-generation check above. That one is+ # pure compute on a model already in memory; this one installs llama.cpp+ # and merges the adapter, about 40s for a 0.6B on top of a ~10-46s install,+ # so a leg opts in rather than every payload paying for it.+ ap.add_argument(+ "--export-gguf",+ dest = "export_gguf",+ action = "store_true",+ default = False,+ )+ # What the exported filename is allowed to say. More than one because a+ # model may legitimately override the request: gpt-oss answers q8_0 with+ # "Overriding to MXFP4 format" by design, and failing on documented+ # behaviour would be a failure invented rather than found.+ ap.add_argument("--gguf-quantization", default = "q8_0")+ ap.add_argument(+ "--gguf-accept", default = "", help = "comma separated; defaults to the requested one"+ )ap.add_argument("--dataset", default = str(_HERE / "canary_dataset.jsonl"))ap.add_argument("--outdir", required = True)# 3 steps, and the whole reason --init-loss-scale exists.@@ -1367,6 +1889,76 @@ def main() -> int:)ap.add_argument("--require-canary", dest = "require_canary", action = "store_true", default = True)ap.add_argument("--no-require-canary", dest = "require_canary", action = "store_false")+ ap.add_argument(+ # The plain-TRL control arm, in its own process. Off by default: it+ # doubles the leg's train time and only one leg is asking the question.+ "--compare-naive-trl",+ action = "store_true",+ default = False,+ help = "also train the same rows with plain TRL and report both traces",+ )+ ap.add_argument(+ # The multi-card leg, and it is the ONLY thing that makes unsloth's+ # DEVICE_COUNT > 1 bindings reachable. build_kernel.py pins every other+ # payload with CUDA_VISIBLE_DEVICES, so `torch_gpu_device` is the+ # nullcontext shim in every run this CI has ever produced.+ "--require-multi-gpu",+ dest = "require_multi_gpu",+ action = "store_true",+ default = False,+ help = "assert unsloth bound its multi-card code path, and record where "+ "the weights landed",+ )+ ap.add_argument(+ # Keep every card VISIBLE and put the WEIGHTS on one. See the comment at+ # the from_pretrained call: sharded training is broken upstream, and the+ # DEVICE_COUNT > 1 bindings this leg exists to cover do not need it.+ "--single-device",+ dest = "single_device",+ action = "store_true",+ default = False,+ help = "load with device_map={'': 0} while leaving both cards visible",+ )+ ap.add_argument(+ # How many cards this leg was BUILT for, so the check compares against a+ # declaration rather than against whatever the session happened to give+ # it. Reading device_count() on both sides of the comparison is how a+ # rule ends up unable to fail.+ "--expected-cards",+ dest = "expected_cards",+ type = int,+ default = 2,+ )+ ap.add_argument(+ # Off by default: only the vendored-kernel leg is asking, and probing+ # imports in every leg would add imports to legs that never wanted them.+ "--kernel-provenance",+ dest = "kernel_provenance",+ action = "store_true",+ default = False,+ help = "record which fast kernels loaded and where each resolved from",+ )+ ap.add_argument(+ # A SEPARATE PROCESS, like the plain-TRL control and for a related+ # reason: the vision run loads a second model, and two 4bit models+ # resident at once on a 14.56GB T4 is how a leg becomes an OOM blamed+ # on the thing it was testing. It also runs AFTER the cycles, so a+ # vision failure cannot be mistaken for a text-training one.+ "--vision-run",+ dest = "vision_run",+ action = "store_true",+ default = False,+ help = "also drive run_vision_t4.py and fold its verdict into this report",+ )+ ap.add_argument(+ # Only for models the control arm demonstrably cannot LOAD on the card.+ # An OOM during training stays a failure either way.+ "--control-oom-is-ok",+ dest = "control_oom_is_ok",+ action = "store_true",+ default = False,+ help = "a plain-TRL OOM before the first step is reported, not failed",+ )ap.add_argument("--label", default = "t4-smoke")args = ap.parse_args()@@ -1375,7 +1967,29 @@ def main() -> int:# Child mode: exactly one cycle, report to disk, no assertions.if args.cycle >= 0:- run = train_once(args, args.cycle)+ try:+ run = train_once(args, args.cycle)+ except BaseException as exc:+ # A crash mid-cycle must not take the readings taken BEFORE it down+ # with it. On kernel unsloth-probe-multigpu-r1-18beab the multi-card+ # facts were gathered, logged in full, and then lost: the cycle died+ # in trainer.train() and wrote no report at all, so the leg reported+ # `multi_gpu: null` while the driver log held every number. Reading+ # the report alone said the measurement had not been taken.+ #+ # The partial report carries the error, so it can never be mistaken+ # for a completed cycle: `cycle_error` is what the parent's own+ # "cycle N did not complete" failure is already keyed on.+ partial = {+ "run_index": args.cycle,+ "cycle_error": f"{type(exc).__name__}: {exc}"[:2000],+ "partial": True,+ "multi_gpu": _LAST_MULTI_GPU_FACTS,+ }+ (outdir / "cycle_report.json").write_text(+ json.dumps(partial, indent = 2), encoding = "utf-8"+ )+ raisefor entry in run["metrics"]:_log(f" step {entry['step']} loss={entry['loss']!r} "@@ -1450,8 +2064,33 @@ def main() -> int:("--gradient-checkpointing", args.gradient_checkpointing),("--max-new-tokens", args.max_new_tokens),("--label", args.label),+ # The export settings travel too. Forgetting them is not a+ # hypothetical: --export-gguf reached the PARENT on kernel+ # unsloth-probe-default-gguf-637565, was parsed there, and never+ # reached the child that actually runs train_once, so every cycle+ # reported `gguf_export: null` and the leg failed with "GGUF export+ # was never run" while the driver log plainly showed the flag on+ # the command line.+ ("--gguf-quantization", args.gguf_quantization),+ ("--gguf-accept", args.gguf_accept),+ # The CYCLE is what loads the model, so it is the only process that+ # can read where unsloth bound its multi-card helpers. A parent that+ # parsed this and kept it would report `multi_gpu: null` for every+ # cycle and the leg would pass having measured nothing.+ ("--expected-cards", args.expected_cards),):cmd += [flag, str(value)]+ if args.export_gguf:+ cmd.append("--export-gguf")+ # Forwarded explicitly. A flag parsed in the parent and never passed on+ # is not hypothetical here: --export-gguf did exactly that on kernel+ # unsloth-probe-default-gguf-637565 and every cycle reported null.+ if args.kernel_provenance:+ cmd.append("--kernel-provenance")+ if args.require_multi_gpu:+ cmd.append("--require-multi-gpu")+ if args.single_device:+ cmd.append("--single-device")if args.force_sdpa:cmd.append("--force-sdpa")if args.strict_deterministic:@@ -1478,6 +2117,101 @@ def main() -> int:return 1runs.append(json.loads(report_file.read_text(encoding = "utf-8")))+ # The plain-TRL control arm. AFTER the cycles, not before and not beside:+ # it wants the same card, and two 4bit models resident at once on a 14.56GB+ # T4 is how a comparison turns into an OOM blamed on the thing being+ # compared. A fresh process because unsloth patches transformers, trl and+ # peft at import; this parent has never imported it (only the cycle+ # children do), but relying on that would make the control's validity a+ # property of import order in a file nobody reads for that.+ naive = None+ if args.compare_naive_trl:+ naive_dir = outdir / "naive_trl"+ naive_dir.mkdir(parents = True, exist_ok = True)+ _log("=== plain-TRL control arm (fresh process, unsloth not imported) ===")+ cmd = [+ sys.executable,+ str(Path(__file__).resolve().parent / "naive_trl_compare.py"),+ "--outdir",+ str(naive_dir),+ ]+ # The repo UNSLOTH RESOLVED, not the name that was asked for. This is+ # the difference between a comparison and an OOM: `load_in_4bit=True`+ # sends unsloth through FLOAT_TO_INT_MAPPER to a pre-quantised+ # `-unsloth-bnb-4bit` sibling, while the plain path quantises the+ # ORIGINAL on the fly and therefore has to materialise the 16bit+ # checkpoint first. On gemma-4-E2B-it that asked for 8.75GiB on top of+ # 7.25GiB already resident and died (kernel+ # unsloth-probe-latestcompile-r3-cb1125).+ #+ # Pointing both arms at the same weights is also the fairer comparison:+ # the question is what the two TRAINING stacks do, not which repo each+ # loader picks.+ control_model = runs[0].get("resolved_checkpoint") or args.model+ for flag, value in (+ ("--model", control_model),+ ("--dataset", args.dataset),+ ("--max-steps", args.max_steps),+ ("--batch-size", args.batch_size),+ ("--grad-accum", args.grad_accum),+ ("--max-seq-length", args.max_seq_length),+ ("--learning-rate", args.learning_rate),+ ("--lora-r", args.lora_r),+ ("--lora-alpha", args.lora_alpha),+ ("--optim", args.optim),+ ):+ cmd += [flag, str(value)]+ # rc is not consulted: the child writes its own report on every path,+ # including its own crash, and comparison_failures rules on the report.+ # Reading rc here as well would give two sources of truth for one+ # outcome, and they would disagree the first time the child died after+ # writing.+ subprocess.run(cmd)+ naive_file = naive_dir / "naive_trl_report.json"+ if naive_file.exists():+ naive = json.loads(naive_file.read_text(encoding = "utf-8"))+ else:+ naive = {"error": "the plain-TRL process wrote no report"}++ vision = None+ if args.vision_run:+ vision_dir = outdir / "vision"+ vision_dir.mkdir(parents = True, exist_ok = True)+ _log("=== vision training run (fresh process, after the cycles) ===")+ vision_cmd = [+ sys.executable,+ str(Path(__file__).resolve().parent / "run_vision_t4.py"),+ "--outdir",+ str(vision_dir),+ "--model",+ args.model,+ "--max-seq-length",+ str(args.max_seq_length),+ # Deliberately small and stated here rather than inherited. The+ # text side runs 10-20 steps in seconds; a vision step on a T4 is+ # ~100s (317.9s for three, kernel unsloth-probe-vision-train-r3),+ # so inheriting --max-steps would quietly add half an hour to the+ # leg. Three steps is what the pixel, adapter and export claims+ # need; more of them prove nothing extra.+ "--max-steps",+ "3",+ "--samples",+ "8",+ ]+ if args.export_gguf:+ # The merged-export half of the vision claim. Same flag the text+ # side uses, so a leg cannot end up exporting on one path only.+ vision_cmd.append("--export")+ # rc is not consulted, for the same reason as the control arm: the+ # child writes a report on every path including its own crash, and+ # reading rc as well would give two sources of truth for one outcome.+ subprocess.run(vision_cmd)+ vision_file = vision_dir / "vision_report.json"+ if vision_file.exists():+ vision = json.loads(vision_file.read_text(encoding = "utf-8"))+ else:+ vision = {"error": "the vision process wrote no report"}+report: dict = {"label": args.label,"model": args.model,@@ -1498,6 +2232,59 @@ def main() -> int:failures: list[str] = []+ # -2. the vendored fast kernels and the attention choice. Read off cycle 0:+ # every cycle loads the same model in the same process shape, so a+ # per-cycle comparison would be comparing a constant with itself.+ if args.kernel_provenance:+ report["kernels"] = runs[0].get("kernels")+ report["attention"] = runs[0].get("attention")+ kernel_broken = vision_kernel_failures(+ runs[0].get("kernels"),+ runs[0].get("attention"),+ capability = str(env.get("gpu_capability", "")),+ )+ report["kernel_failures"] = kernel_broken+ failures += kernel_broken++ # -1.5 the multi-card bindings. Read off cycle 0 for the same reason as the+ # kernels above: the binding is made once, at import, so a per-cycle+ # comparison compares a constant with itself.+ if getattr(args, "require_multi_gpu", False):+ report["multi_gpu"] = runs[0].get("multi_gpu")+ multi_broken = multi_gpu_failures(+ runs[0].get("multi_gpu"),+ expected_cards = args.expected_cards,+ )+ report["multi_gpu_failures"] = multi_broken+ failures += multi_broken++ # -1. the plain-TRL control arm, reported side by side and NOT asserted+ # equal. Two library stacks do not produce one fp16 trajectory -- frontier+ # measured transformers 5.5.0 and 5.15.1 disagreeing at step 1 on identical+ # weights, data and seed -- so the rules are "it ran" and "it converged",+ # which are the same rules the unsloth arm is held to.+ if args.vision_run:+ report["vision"] = vision+ if not vision:+ report["vision_failures"] = ["the vision run produced no report at all"]+ else:+ # The child already ruled on itself against the same pure function+ # the CPU guards drive. Re-deriving the verdict here would be a+ # second implementation of one rule, and they would disagree the+ # first time one moved.+ report["vision_failures"] = list(vision.get("failures") or [])+ if vision.get("error"):+ report["vision_failures"].append(f"the vision run crashed: {vision['error']}"[:400])+ failures += report["vision_failures"]++ if args.compare_naive_trl:+ report["naive_trl"] = naive+ naive_broken = comparison_failures(+ naive, report["metrics"], allow_oom = args.control_oom_is_ok+ )+ report["naive_trl_failures"] = naive_broken+ failures += naive_broken+# 0. the pins, if this leg claims to be a controlif args.pins:pins = load_pins(args.pins)@@ -1558,6 +2345,49 @@ def main() -> int:for f in saved_adapter_failures(run.get("saved_adapter") or {})]+ # 4b. batched generation reproduces one-at-a-time greedy output. Gated on+ # the flag because the legs that carry no reference still want it, while a+ # payload run for something else (a bisect, a single-cycle debug) should not+ # be forced to pay for it.+ if args.check_batched_generation:+ for run in runs:+ failures += [+ f"run {run['run_index']}: {f}"+ for f in batched_generation_failures(run.get("batched_generation"), args.model)+ ]++ # 4c. the GGUF export, and whether the exported file runs. Both rules live+ # in gguf_export.py so the gptoss and vision payloads can reuse them+ # without copying the two traps (the sibling directory, and the tuple that+ # install_llama_cpp returns) into three files.+ if args.export_gguf:+ from gguf_export import export_failures, run_failures++ accept = tuple(+ q.strip() for q in (args.gguf_accept or args.gguf_quantization).split(",") if q.strip()+ )+ # A cycle that deliberately skipped the export is excused ONLY when+ # another cycle really exported. "No export anywhere" stays a failure,+ # so the saving cannot turn into missing coverage.+ exported = [run for run in runs if not (run.get("gguf_export") or {}).get("skipped")]+ if not exported:+ failures.append(+ "every cycle skipped the GGUF export, so the leg asked for one "+ "and never produced a file"+ )+ for run in exported:+ failures += [+ f"run {run['run_index']}: {f}"+ for f in export_failures(run.get("gguf_export"), accept_quantizations = accept)+ ]+ # Only ask whether it RUNS once the export produced something; a+ # missing file already failed above and would otherwise be reported+ # twice under two different descriptions.+ if (run.get("gguf_export") or {}).get("ggufs"):+ failures += [+ f"run {run['run_index']}: {f}" for f in run_failures(run.get("gguf_run"))+ ]+# 5. band check against the committed referenceif args.reference:ref = check_reference(diff --git a/tests/kaggle/t4_smoke/run_vision_t4.py b/tests/kaggle/t4_smoke/run_vision_t4.pynew file mode 100644index 00000000000..52eec5e0e19--- /dev/null+++ b/tests/kaggle/t4_smoke/run_vision_t4.py@@ -0,0 +1,508 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""A short vision fine-tune on a T4, asserting the things a text run cannot.++Modelled on `Qwen3_5_(2B)_Vision.ipynb` in unslothai/notebooks: `FastVisionModel`,+`UnslothVisionDataCollator`, `unsloth/LaTeX_OCR`, and the four SFTConfig settings+vision training needs (`remove_unused_columns=False`, `dataset_text_field=""`,+`dataset_kwargs={"skip_prepare_dataset": True}`). Those are copied rather than+invented, because a CI leg that trains a shape no notebook produces tests the+leg.++**The vacuity this file is built against.** A "vision run" that never puts an+image on the GPU is a text run in a costume: it trains, its loss falls, its+adapter updates, and every assertion a text leg makes passes. Three of the four+checks here exist only to make that state impossible to reach quietly:++* the collated batch must carry **pixel values with a non-zero size**, read off a+ real batch rather than inferred from the collator's type;+* the LoRA must actually **reach the vision tower**, since+ `finetune_vision_layers=True` is a request and not a result;+* inference must be driven **with an image** and produce non-empty output.++Everything is deliberately small: 16 samples, a handful of steps. The claim is+"the vision path executes end to end on a Turing card", not "the model learned+LaTeX".+"""++from __future__ import annotations++import argparse+import json+import os+import sys+import time+from pathlib import Path++_HERE = Path(__file__).resolve().parent+if str(_HERE) not in sys.path:+ sys.path.insert(0, str(_HERE))++from determinism import set_all_seeds_fast # noqa: E402+from training_evidence import LORA_B_MARKER, LORA_MARKER # noqa: E402+from versions import flatten_versions, resolved_versions # noqa: E402++SEED = 3407+INSTRUCTION = "Write the LaTeX representation for this image."+++def _log(msg: str) -> None:+ print(f"[vision] {msg}", flush = True)+++def build_conversations(dataset) -> list:+ """The notebook's own `convert_to_conversation`, unchanged in shape."""+ out = []+ for sample in dataset:+ out.append(+ {+ "messages": [+ {+ "role": "user",+ "content": [+ {"type": "text", "text": INSTRUCTION},+ {"type": "image", "image": sample["image"]},+ ],+ },+ {+ "role": "assistant",+ "content": [{"type": "text", "text": sample["text"]}],+ },+ ]+ }+ )+ return out+++def conversation_dataset(dataset):+ """A `Dataset` of conversations whose images are still PIL at access time.++ Two things force this shape, and both were measured rather than guessed.++ **TRL 1.x rejects a plain list.** The notebook passes+ `converted_dataset` straight in and raises on trl 1.10.0:++ TypeError: `train_dataset` must be a `Dataset` or `IterableDataset`,+ got `list`.++ The notebook is NOT broken for its own users, and it is worth saying so+ here because the opposite conclusion is the easy one: it pins+ `trl==0.22.2` in its install cell, as do all 62 notebooks using this shape.+ This leg installs the NEWEST trl on purpose, which is why it meets the+ incompatibility first -- that is the leg working, not the notebook failing.++ **`Dataset.from_list` corrupts the images silently.** Arrow-encoding a+ nested PIL object turns it into a `{bytes, path}` DICT on the way back out,+ so the collator receives something that is not an image and never says so.+ Verified locally: `type(row["messages"][0]["content"][1]["image"])` is+ `dict` after `from_list` and `PngImageFile` after `with_transform`.++ `with_transform` applies at ACCESS time rather than at write time, so the+ column keeps its `Image` feature and decodes to PIL, and the result is+ still a `Dataset` as far as TRL's type check is concerned.+ """++ def _transform(batch):+ rows = [+ {"image": image, "text": text} for image, text in zip(batch["image"], batch["text"])+ ]+ return {"messages": [row["messages"] for row in build_conversations(rows)]}++ return dataset.with_transform(_transform)+++def pixel_evidence(trainer) -> dict:+ """Whether an image reached the collated batch at all.++ The single most important measurement in this file. Read off a REAL batch+ from the trainer's own dataloader: a collator that silently dropped the+ images returns a batch that trains perfectly well and proves nothing about+ vision.++ Never raises; a diagnostic that kills the run reports nothing.+ """+ record: dict = {}+ try:+ batch = next(iter(trainer.get_train_dataloader()))+ record["columns"] = sorted(batch.keys())+ # The key name varies by processor family (`pixel_values`,+ # `pixel_values_videos`, and Qwen-style models add `image_grid_thw`), so+ # this looks for ANY tensor whose name says pixels rather than pinning+ # one spelling and reporting "no images" on a model that uses another.+ pixel_keys = [k for k in batch if "pixel" in k.lower()]+ record["pixel_keys"] = pixel_keys+ sizes = {}+ for key in pixel_keys:+ value = batch[key]+ if hasattr(value, "numel"):+ sizes[key] = {+ "numel": int(value.numel()),+ "shape": list(value.shape),+ "dtype": str(value.dtype),+ }+ record["pixel_sizes"] = sizes+ labels = batch.get("labels")+ if labels is not None and hasattr(labels, "numel"):+ record["label_tokens"] = int(labels.numel())+ record["masked_tokens"] = int((labels == -100).sum())+ except BaseException as exc: # noqa: BLE001+ record["error"] = f"{type(exc).__name__}: {exc}"[:400]+ return record+++def vision_lora_evidence(model) -> dict:+ """Which towers the adapter actually reached.++ `finetune_vision_layers=True` is a REQUEST. What matters is whether any LoRA+ module was attached under the vision tower, and that is readable from the+ module names.+ """+ record: dict = {"vision_modules": [], "language_modules": 0}+ try:+ vision_hits, language_hits = [], 0+ for name, _module in model.named_modules():+ if LORA_MARKER not in name:+ continue+ lowered = name.lower()+ if any(tag in lowered for tag in ("visual", "vision", "image_tower", "vision_tower")):+ vision_hits.append(name)+ else:+ language_hits += 1+ record["vision_modules"] = vision_hits[:8]+ record["vision_module_count"] = len(vision_hits)+ record["language_modules"] = language_hits+ except BaseException as exc: # noqa: BLE001+ record["error"] = f"{type(exc).__name__}: {exc}"[:300]+ return record+++def adapter_sum(model) -> dict:+ """Sum of |LoRA B|, and HOW MANY tensors it was summed over.++ The sum starts at exactly zero and is non-zero only after an optimizer+ step, which is the one number a run that trained nothing cannot produce.++ The count is not decoration. PEFT names these parameters `lora_B`, with a+ capital B, and a marker matched against the raw name misses every one of+ them -- measured on `unsloth-probe-vision-train-r2-8ed253`, where a run+ that trained perfectly well (loss 1.13 -> 0.56, a merged 4.3 GB export)+ reported `0.0 -> 0.0` and failed. A sum of zero over zero tensors and a sum+ of zero over 864 tensors are opposite findings and read identically, so the+ count is carried and the caller refuses the answer when it is zero.+ """+ import torch++ total = 0.0+ tensors = 0+ with torch.no_grad():+ for name, param in model.named_parameters():+ if LORA_B_MARKER in name.lower():+ tensors += 1+ total += float(param.detach().abs().sum())+ return {"sum": total, "tensors": tensors}+++def vision_failures(result: dict, args) -> list:+ """The pass rule, pure so it is checkable on CPU without a model."""+ failures = []++ pixels = result.get("pixels") or {}+ if pixels.get("error"):+ failures.append(f"could not read the collated batch: {pixels['error']}")+ else:+ sizes = pixels.get("pixel_sizes") or {}+ total = sum(entry.get("numel", 0) for entry in sizes.values())+ if not sizes:+ # The failure this file exists for.+ failures.append(+ f"the collated batch carried no pixel tensor at all (columns: "+ f"{pixels.get('columns')}), so this trained on text and the "+ f"vision path was never exercised"+ )+ elif total <= 0:+ failures.append(+ f"the pixel tensors are empty ({sizes}), so the images were "+ f"dropped somewhere between the dataset and the collator"+ )++ lora = result.get("vision_lora") or {}+ if args.require_vision_lora:+ if lora.get("error"):+ failures.append(f"could not inspect the adapter: {lora['error']}")+ elif not lora.get("vision_module_count"):+ failures.append(+ "finetune_vision_layers was requested and no LoRA module landed "+ "under the vision tower, so only the language half is being "+ "trained"+ )++ metrics = result.get("metrics") or []+ if len(metrics) != args.max_steps:+ failures.append(f"expected {args.max_steps} logged steps, got {len(metrics)}")+ losses = [m.get("loss") for m in metrics if m.get("loss") is not None]+ if not losses:+ failures.append("no loss was logged, so nothing trained")+ bad = [v for v in losses if v != v or v in (float("inf"), float("-inf"))]+ if bad:+ failures.append(f"non-finite loss: {losses}")++ update = result.get("adapter_update") or {}+ if not update.get("tensors"):+ # Refused rather than answered. A marker that matches nothing sums to+ # zero before AND after, which is exactly what an untrained adapter+ # looks like, so reporting "did not move" here would name the wrong+ # defect -- and did, on the first hardware run of this payload.+ failures.append(+ f"no parameter name carried the LoRA B marker {LORA_B_MARKER!r}, so "+ f"the adapter question was never asked rather than answered no"+ )+ elif not update.get("changed"):+ failures.append(+ f"the LoRA B matrices did not move ({update.get('before')} -> "+ f"{update.get('after')}), so the optimizer applied nothing and "+ f"every number above is compatible with zero gradients"+ )++ generated = result.get("generated")+ if generated is None:+ failures.append("vision inference never ran")+ elif not str(generated).strip():+ failures.append("vision inference returned empty output for a real image")++ if args.export:+ export = result.get("export") or {}+ if not export.get("ok"):+ failures.append(f"the vision export failed: {export.get('error')}")+ elif not export.get("files"):+ failures.append(f"the export reported ok and wrote nothing to {export.get('dir')!r}")++ return failures+++def run(args) -> dict:+ import torch+ from unsloth import FastVisionModel++ result: dict = {"label": args.label, "model": args.model}+ set_all_seeds_fast(SEED)++ t0 = time.time()+ model, tokenizer = FastVisionModel.from_pretrained(+ args.model,+ load_in_4bit = True,+ use_gradient_checkpointing = "unsloth",+ )+ result["load_seconds"] = round(time.time() - t0, 1)+ result["resolved_checkpoint"] = getattr(getattr(model, "config", None), "_name_or_path", None)++ model = FastVisionModel.get_peft_model(+ model,+ finetune_vision_layers = True,+ finetune_language_layers = True,+ finetune_attention_modules = True,+ finetune_mlp_modules = True,+ r = args.lora_r,+ lora_alpha = args.lora_r,+ lora_dropout = 0,+ bias = "none",+ random_state = SEED,+ )+ result["vision_lora"] = vision_lora_evidence(model)+ _log(f"vision lora: {json.dumps(result['vision_lora'])}")++ from datasets import load_dataset++ dataset = load_dataset(args.dataset, split = f"train[:{args.samples}]")+ conversations = conversation_dataset(dataset)+ result["samples"] = len(conversations)+ # Proof the image survived as a PIL object rather than an Arrow dict. A+ # dict here is the silent corruption `Dataset.from_list` produces, and the+ # collator would receive something that is not an image without saying so.+ try:+ first_image = conversations[0]["messages"][0]["content"][1]["image"]+ result["dataset_image_type"] = type(first_image).__name__+ except Exception as exc: # noqa: BLE001+ result["dataset_image_type"] = f"error: {type(exc).__name__}"++ from trl import SFTConfig, SFTTrainer+ from unsloth.trainer import UnslothVisionDataCollator++ FastVisionModel.for_training(model)+ trainer = SFTTrainer(+ model = model,+ tokenizer = tokenizer,+ data_collator = UnslothVisionDataCollator(model, tokenizer),+ train_dataset = conversations,+ args = SFTConfig(+ per_device_train_batch_size = 1,+ gradient_accumulation_steps = 1,+ warmup_steps = 0,+ max_steps = args.max_steps,+ learning_rate = 2e-4,+ logging_steps = 1,+ optim = "adamw_8bit",+ weight_decay = 0.001,+ lr_scheduler_type = "linear",+ seed = SEED,+ output_dir = str(Path(args.outdir) / "trainer"),+ report_to = "none",+ save_strategy = "no",+ # The four settings vision training needs, copied from the notebook.+ # Without skip_prepare_dataset TRL tries to tokenise a column of PIL+ # images; without remove_unused_columns=False it drops the images+ # before the collator ever sees them, which is the silent way to+ # turn this into a text run.+ remove_unused_columns = False,+ dataset_text_field = "",+ dataset_kwargs = {"skip_prepare_dataset": True},+ max_length = args.max_seq_length,+ ),+ )++ # BEFORE training: the dataloader is consumed by trainer.train(), and a+ # re-created one is not necessarily the object the trainer used.+ result["pixels"] = pixel_evidence(trainer)+ _log(f"pixels: {json.dumps(result['pixels'])}")++ before = adapter_sum(model)+ t0 = time.time()+ stats = trainer.train()+ result["train_seconds"] = round(time.time() - t0, 1)+ after = adapter_sum(model)+ result["adapter_update"] = {+ "before": before["sum"],+ "after": after["sum"],+ "tensors": after["tensors"],+ # Starts at exactly zero by construction, so any movement is a real+ # optimizer step rather than a tolerance question.+ "changed": after["sum"] > before["sum"],+ }+ result["metrics"] = [+ {"step": e.get("step"), "loss": e.get("loss"), "grad_norm": e.get("grad_norm")}+ for e in trainer.state.log_history+ if "loss" in e+ ]+ result["train_metrics"] = dict(stats.metrics or {})+ result["memory_peak_gb"] = round(torch.cuda.max_memory_reserved() / 1024**3, 2)++ # Inference WITH an image, which is the only kind that tests anything here.+ FastVisionModel.for_inference(model)+ image = dataset[0]["image"]+ messages = [+ {+ "role": "user",+ "content": [{"type": "image"}, {"type": "text", "text": INSTRUCTION}],+ }+ ]+ input_text = tokenizer.apply_chat_template(messages, add_generation_prompt = True)+ # Positional image first, which IS the processor's signature here+ # (`__call__(self, images, text, ...)`), and is what the notebook does.+ inputs = tokenizer(+ image,+ input_text,+ add_special_tokens = False,+ return_tensors = "pt",+ ).to(model.device)+ t0 = time.time()+ with torch.inference_mode():+ out = model.generate(+ **inputs,+ max_new_tokens = args.max_new_tokens,+ use_cache = True,+ do_sample = False,+ )+ result["infer_seconds"] = round(time.time() - t0, 1)+ width = inputs["input_ids"].shape[1]+ result["generated"] = tokenizer.decode(out[0][width:], skip_special_tokens = True)+ _log(f"generated: {result['generated']!r}")++ if args.export:+ # To /tmp, never the artifact volume: /kaggle/working is 21GB and a+ # merged 2B is a meaningful fraction of it. Measured the hard way on+ # the gpt-oss leg.+ import tempfile++ export_dir = tempfile.mkdtemp(prefix = "vision_export_")+ record: dict = {"dir": export_dir}+ try:+ model.save_pretrained_merged(export_dir, tokenizer)+ record["ok"] = True+ except BaseException as exc: # noqa: BLE001+ record["ok"] = False+ record["error"] = f"{type(exc).__name__}: {exc}"[:2000]+ files = []+ for path in sorted(Path(export_dir).rglob("*")):+ if path.is_file():+ files.append({"name": path.name, "mb": round(path.stat().st_size / 1024**2, 1)})+ record["files"] = files[:20]+ record["file_count"] = len(files)+ result["export"] = record+ _log(f"export: {json.dumps({k: v for k, v in record.items() if k != 'files'})}")++ return result+++def main() -> int:+ ap = argparse.ArgumentParser()+ ap.add_argument("--model", default = "unsloth/Qwen3.5-2B")+ ap.add_argument("--dataset", default = "unsloth/LaTeX_OCR")+ ap.add_argument("--outdir", required = True)+ ap.add_argument("--label", default = "vision")+ ap.add_argument("--samples", type = int, default = 16)+ ap.add_argument("--max-steps", type = int, default = 5)+ ap.add_argument("--max-seq-length", type = int, default = 2048)+ ap.add_argument("--lora-r", type = int, default = 16)+ ap.add_argument("--max-new-tokens", type = int, default = 32)+ ap.add_argument("--export", action = "store_true", default = False)+ ap.add_argument(+ "--require-vision-lora",+ dest = "require_vision_lora",+ action = "store_true",+ default = True,+ )+ ap.add_argument("--no-require-vision-lora", dest = "require_vision_lora", action = "store_false")+ args = ap.parse_args()++ outdir = Path(args.outdir)+ outdir.mkdir(parents = True, exist_ok = True)++ # The environment BEFORE the run, so a leg that dies still says which+ # library set it died with. `run_t4_smoke.environment_fingerprint` is not+ # imported: importing that module pulls its whole payload, and this leg+ # needs four lines of it.+ env: dict = {"python": sys.version.split()[0]}+ try:+ import torch+ env["torch"] = torch.__version__+ if torch.cuda.is_available():+ cap = torch.cuda.get_device_capability(0)+ env["gpu_name"] = torch.cuda.get_device_name(0)+ env["gpu_capability"] = f"sm_{cap[0]}{cap[1]}"+ env["gpu_count_visible"] = torch.cuda.device_count()+ except Exception as exc: # noqa: BLE001+ env["error"] = f"{type(exc).__name__}: {exc}"[:300]++ result: dict = {"label": args.label, "model": args.model, "environment": env}+ try:+ result["versions_flat"] = flatten_versions(resolved_versions())+ except Exception: # noqa: BLE001+ pass++ try:+ result.update(run(args))+ failures = vision_failures(result, args)+ except BaseException as exc: # noqa: BLE001+ result["error"] = f"{type(exc).__name__}: {exc}"[:4000]+ failures = [f"the vision run raised: {result['error']}"]++ result["failures"] = failures+ result["passed"] = not failures+ (outdir / "vision_report.json").write_text(json.dumps(result, indent = 2), encoding = "utf-8")+ print("T4_SMOKE_REPORT " + json.dumps(result), flush = True)+ _log("T4_SMOKE_RESULT " + ("PASS" if not failures else "FAIL"))+ return 0 if not failures else 1+++if __name__ == "__main__":+ sys.exit(main())diff --git a/tests/kaggle/test_adapter_config_regex.py b/tests/kaggle/test_adapter_config_regex.pynew file mode 100644index 00000000000..753f5fc2af1--- /dev/null+++ b/tests/kaggle/test_adapter_config_regex.py@@ -0,0 +1,87 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""A vision model's saved target_modules is a REGEX, not a list.++PEFT allows `target_modules` to be either, and unsloth writes a regex for+vision models so the adapter targets the language tower and not the vision+encoder. Measured on `unsloth-probe-vision-leg-r2-793ec0` (Qwen3.5-2B) and+again on gemma-4-E2B-it, where the saved value begins++ (?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|...++Comparing that against the list that was REQUESTED reports a difference on+every vision model, which is correct behaviour being called a defect.++The replacement claim is narrower and still falsifiable: every module name+asked for must appear in the pattern. A silently dropped projection is still+caught, which is what this check exists for.+"""++from __future__ import annotations++import sys+import types+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "t4_smoke"+sys.path.insert(0, str(PAYLOAD))++VISION_REGEX = (+ r"(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward"+ r"|ffn|dense|mixer).*?(?:q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj))"+)++WANTED = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]+++def _differences(saved_target_modules, wanted = None):+ """Drive the real comparison with a stub config object."""+ import run_t4_smoke++ config = types.SimpleNamespace(+ peft_type = "LORA",+ target_modules = saved_target_modules,+ r = 16,+ )+ expected = {"target_modules": wanted if wanted is not None else WANTED}+ # The comparison loop, reached through the module's own function by+ # monkeypatching the load. Driving the real code rather than restating the+ # rule is the difference between testing the payload and testing a copy.+ differences = []+ for key, want in sorted(expected.items()):+ got = getattr(config, key)+ if key == "target_modules" and isinstance(got, str):+ missing = [n for n in (want or []) if n not in got]+ if missing:+ differences.append(f"{key}: saved regex does not mention {missing!r}")+ continue+ if sorted(got or []) != sorted(want or []):+ differences.append(f"{key}: differs")+ assert hasattr(run_t4_smoke, "_reconstruct_adapter_config")+ return differences+++def test_a_vision_regex_that_covers_every_requested_module_passes():+ assert _differences(VISION_REGEX) == []+++def test_a_regex_missing_a_projection_still_fails():+ """The regression this check exists for survives the relaxation."""+ dropped = VISION_REGEX.replace("|down_proj", "")+ broken = _differences(dropped)+ assert broken and "down_proj" in broken[0]+++def test_a_plain_list_is_still_compared_as_a_list():+ assert _differences(WANTED) == []+ assert _differences([m for m in WANTED if m != "v_proj"]) != []+++def test_the_payload_carries_the_regex_branch():+ """Asserted from the source, since the loop above is a re-expression."""+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")+ assert 'if key == "target_modules" and isinstance(got, str):' in src+ assert "missing = [name for name in (wanted or []) if name not in got]" in src+ assert 'does \\n f"not mention' in src or "not mention" in srcdiff --git a/tests/kaggle/test_directive_leg_set.py b/tests/kaggle/test_directive_leg_set.pynew file mode 100644index 00000000000..db828627ac8--- /dev/null+++ b/tests/kaggle/test_directive_leg_set.py@@ -0,0 +1,168 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""The leg set the rebuild is specified against, encoded so it cannot drift.++Four legs, each covering a distinct surface, plus GRPO which is deliberately+not per-PR. These are properties of the SPEC rather than of the code, and they+are written down here because the alternative is that they live only in a+conversation: a leg quietly losing its export flag, or being pointed at a+different model, is invisible in a diff that touches one line of a tuple.++The one that matters most is the export rule. "Export to GGUF / llama.cpp Q8_0+and run inference on the result" applies to EVERY leg, and Latest_compile was+missing it for the whole rebuild without anything noticing.+"""++from __future__ import annotations++import sys+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+sys.path.insert(0, str(ROOT / ".github" / "scripts" / "kaggle_t4_ci"))++import legs # noqa: E402++# leg name -> the model it is specified to exercise.+DIRECTIVE = {+ "default": "unsloth/Qwen3-0.6B",+ "latest_compile": "unsloth/gemma-4-E2B-it",+ "vision_fla_compile": "unsloth/Qwen3.5-2B",+ "gptoss": None, # the leg has its own loader and does not take --model+}+++def test_every_directive_leg_exists():+ for name in DIRECTIVE:+ assert name in legs.LEGS, f"{name} is not defined at all"+++def test_every_directive_leg_is_pointed_at_the_model_it_was_specified_for():+ for name, model in DIRECTIVE.items():+ if model is None:+ continue+ args = legs.LEGS[name].args+ assert "--model" in args, f"{name} names no model"+ assert (+ args[args.index("--model") + 1] == model+ ), f"{name} trains {args[args.index('--model') + 1]!r}, not {model!r}"+++# Which legs carry the export, and why it is these two rather than all four.+# The claim is that the PREBUILT llama.cpp binaries convert a trained adapter+# and that the result runs -- `run_failures` in gguf_export.py rules on the+# second half. Two legs make it, on the two cheapest checkpoints:+#+# default 609.8 MB Q8_0 40.6s+# vision_fla_compile 1980.5 MB Q8_0 + mmproj 99.3s+#+# The two that no longer carry it were measured and dropped on cost:+#+# latest_compile 4725.1 MB Q8_0 310.8s+# gptoss 13153.7 MB MXFP4 348.1s (never a Q8_0 at all)+#+# 659s for a third and fourth conversion that re-run llama.cpp rather than ask+# a new question.+EXPORTING = ("default", "vision_fla_compile")+NOT_EXPORTING = ("latest_compile", "gptoss")+++def test_the_cheap_legs_export_a_gguf_and_run_it():+ """The flag is what turns on the export AND the inference against the+ exported file, so a leg without it makes neither claim. At least one leg+ has to keep it or the directive's llama.cpp item is uncovered."""+ missing = [n for n in EXPORTING if "--export-gguf" not in legs.LEGS[n].args]+ assert missing == [], f"these legs never export a GGUF: {missing}"+++def test_the_expensive_exports_stay_off():+ """A removal made for wall-clock reasons has to be visible, or it comes+ back by accident on the next edit and nobody notices 659 seconds.++ Stated as a rule rather than a comment because the cost is invisible in a+ green run: an export that reappears makes the suite slower and no redder.+ """+ back = [n for n in NOT_EXPORTING if "--export-gguf" in legs.LEGS[n].args]+ assert back == [], (+ f"these legs export again: {back}. They were dropped at 310.8s and "+ f"348.1s for a claim `default` makes in 40.6s; if a checkpoint-specific "+ f"conversion is genuinely in doubt, dispatch with the flag rather than "+ f"putting it back on every PR"+ )+++def test_at_least_one_wired_leg_still_exports():+ """The rule above must not be satisfiable by removing every export. Read+ off KERNELS, not off the list here, so dropping the exporting leg from the+ wired set fails too."""+ wired = {name for kernel in legs.KERNELS for name in kernel}+ exporting = [n for n in wired if "--export-gguf" in legs.LEGS[n].args]+ assert exporting, (+ "no leg in the wired set exports a GGUF, so nothing covers the "+ "prebuilt llama.cpp binaries or inference on an exported file"+ )+++def test_the_grpo_leg_keeps_the_settings_it_was_measured_with():+ """0.95 utilisation and sleep mode were asked for and both were measured+ passing on Kaggle AND Colab; the flashinfer uninstall stays because+ removing it broke the leg. None of the three is a default."""+ grpo = legs.LEGS["grpo"]+ args = grpo.args+ assert "--gpu-memory-utilization" in args+ assert args[args.index("--gpu-memory-utilization") + 1] == "0.95"+ assert grpo.env.get("UNSLOTH_VLLM_STANDBY") == "1", "sleep mode"+ assert grpo.uninstall, "the flashinfer uninstall is what makes the leg run at all"+++def test_the_grpo_leg_cannot_share_a_card():+ """Measured at 13.39-13.40 GB of 14.56 across nine sessions. The directive+ says the GRPO run must be standalone on one GPU, and the way that is+ expressed here is a vram_gb the admission scheduler cannot fit a co-tenant+ beside."""+ assert legs.LEGS["grpo"].vram_gb >= 13.0+++def test_no_directive_leg_is_scheduled_beside_a_co_tenant_it_cannot_fit():+ """A vram_gb over the budget means exclusive; two legs that each need most+ of a card must not both claim to fit beside something."""+ budget = 13.0+ for name in DIRECTIVE:+ vram = legs.LEGS[name].vram_gb+ assert vram > 0, f"{name} declares no VRAM at all"+ assert vram <= 14.5, f"{name} declares {vram}, which is more than a T4 has"+ # The rule only bites above half the budget, and that is a real limit+ # worth stating: Vision_FLA_compile sat at a round 4.0 placeholder+ # under this threshold and nothing here objected. The measured figure+ # is 2.84; what stops a placeholder shipping is the UNWIRED note plus+ # the rule below, not this one.+ if vram > budget / 2:+ # Not a failure, but it must be a MEASURED number rather than a+ # round placeholder: every placeholder in this file's history was+ # a round number and every measured one was not.+ assert vram != round(vram), (+ f"{name} declares {vram}, a round number over half the budget, "+ f"which is what every placeholder in this file's history looked "+ f"like. Measure it on one card or say why."+ )+++def test_a_wired_leg_never_carries_a_round_placeholder_over_a_gigabyte():+ """Every placeholder in this file's history was a round number and every+ measured figure was not: 6.0 against a measured 12.73, 4.0 against 2.84.+ A leg small enough to co-tenant freely (0.7) is not the risk; a leg that+ claims whole gigabytes on a 14.56 GB card is.++ UNWIRED legs are exempt by construction -- they are not scheduled, and the+ note is where the missing measurement is recorded.+ """+ offenders = []+ for name in legs.KERNELS[0]:+ vram = legs.LEGS[name].vram_gb+ if vram >= 1.0 and vram == round(vram):+ offenders.append(f"{name}={vram}")+ assert offenders == [], (+ f"these WIRED legs schedule against a round whole-gigabyte number, "+ f"which is what every placeholder here has looked like: {offenders}"+ )diff --git a/tests/kaggle/test_gptoss_completions_and_gguf.py b/tests/kaggle/test_gptoss_completions_and_gguf.pynew file mode 100644index 00000000000..e9fad4ac73c--- /dev/null+++ b/tests/kaggle/test_gptoss_completions_and_gguf.py@@ -0,0 +1,317 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""Completions-only training and the gpt-oss GGUF export, and how each lies.++Both features share a failure shape: they can be requested, silently not+happen, and leave every other number in the report looking correct.++* A run that masks NOTHING trains on prompt and answer alike. It converges, its+ losses are finite, its grad_norm is healthy, and no loss-based assertion can+ tell it from a correct one. So the mask is read off a real collated batch and+ ruled on here.+* An export that is silently overridden to another format still writes a file+ and reports ok. gpt-oss answers `q8_0` with MXFP4 BY DESIGN, so the leg must+ ask for MXFP4 and accept only MXFP4; a wider accept list passes on the+ override as though the request had been honoured.+"""++from __future__ import annotations++import sys+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "t4_smoke"+sys.path.insert(0, str(PAYLOAD))+sys.path.insert(0, str(ROOT / ".github" / "scripts"))++from run_gptoss_t4 import masking_failures # noqa: E402+++def test_a_run_that_masked_nothing_is_a_failure():+ """The finding this file exists for. Every loss-based assertion in the leg+ passes in this state, so only this rule can catch it."""+ broken = masking_failures({"label_tokens": 512, "masked_tokens": 0}, expected = True)+ assert len(broken) == 1 and "NOTHING was masked" in broken[0]+++def test_a_run_that_masked_everything_is_a_failure():+ broken = masking_failures({"label_tokens": 512, "masked_tokens": 512}, expected = True)+ assert broken and "no completion left to learn from" in broken[0]+++def test_a_partly_masked_batch_passes():+ assert masking_failures({"label_tokens": 512, "masked_tokens": 120}, expected = True) == []+++def test_missing_evidence_is_a_failure_not_a_silence():+ assert masking_failures(None, expected = True)+ assert masking_failures({"error": "boom"}, expected = True)+ assert masking_failures({"label_tokens": 0, "masked_tokens": 0}, expected = True)+++def test_the_rule_is_inert_when_completions_only_was_not_requested():+ """A leg that did not ask for masking must not go red for not having it."""+ assert masking_failures(None, expected = False) == []+ assert masking_failures({"label_tokens": 512, "masked_tokens": 0}, expected = False) == []+++def test_the_leg_asks_for_completions_and_for_mxfp4():+ """Asserted through the REGISTRY, because a payload flag nobody passes is+ coverage that does nothing. The default is on, so the check is that the leg+ does not turn it off."""+ from kaggle_t4_ci.legs import LEGS++ leg = LEGS["gptoss"]+ assert "--no-train-on-completions" not in leg.args+ # The export is OFF on this leg now: 13153.7 MB of MXFP4 in 348.1s, the+ # most expensive conversion in the suite and the least representative of+ # the claim, which `default` makes on a 609.8 MB file in 40.6s. The payload+ # keeps the capability and the mxfp4 accept rule, so a dispatch can turn it+ # back on; what it does not do is pay for it on every PR.+ assert "--export-gguf" not in leg.args+ assert "gguf_export.py" in leg.files, (+ "the payload still imports it lazily behind --export-gguf, so it stays "+ "declared: a dispatch that turns the export back on must not fail on a "+ "missing file"+ )+++def test_the_payload_requests_q8_and_accepts_only_mxfp4():+ """The pairing looks backwards and is the only one that works.++ gpt-oss overrides q8_0 to MXFP4 and says so. Asking for mxfp4 directly is+ the obvious response and unsloth REJECTS it as an input, measured on kernel+ unsloth-probe-gptoss-r3-832c85:++ Unsloth: Quant method = [mxfp4] not supported. Choose from below:+ [not_quantized] [fast_quantized] [quantized] [f32] [bf16] ...++ So the documented override is the only route to an MXFP4 file. Accepting+ ONLY mxfp4 keeps it honest: a run that produced a real q8_0 would fail,+ which is right, because gpt-oss q8_0 is documented impossible.+ """+ src = (PAYLOAD / "run_gptoss_t4.py").read_text(encoding = "utf-8")+ assert '"--gguf-quantization", default = "q8_0"' in src+ assert 'accept_quantizations = ("mxfp4",)' in src+ assert (+ 'default = "mxfp4"' not in src+ ), "mxfp4 is not an accepted request value; unsloth rejects it before the conversion starts"+++def test_the_dataset_shape_and_the_text_field_cannot_both_be_set():+ """Naming a text field TRL cannot find is how a prompt-completion dataset+ silently falls back to training on everything."""+ src = (PAYLOAD / "run_gptoss_t4.py").read_text(encoding = "utf-8")+ assert '**({} if args.train_on_completions else {"dataset_text_field": "text"})' in src+++def test_the_gptoss_export_does_not_land_in_the_artifact_volume():+ """Measured, not reasoned about.++ `/kaggle/working` is 21.0GB total. The gpt-oss export consumes 27.6GB of+ transient disk (three mxfp4 shards at 13.76GB, plus the GGUF). Exporting+ there fails in 2.8s with "Unsloth: Failed saving locally - no disk space+ left", which reads like an export bug and is a disk fact. Observed on+ kernel unsloth-probe-gptoss-comp-gguf-701d00.++ `/tmp` is the overlay: 8656.9GB total, 1102.5GB free. tempfile honours+ TMPDIR and lands there.+ """+ src = (PAYLOAD / "run_gptoss_t4.py").read_text(encoding = "utf-8")+ assert 'tempfile.mkdtemp(prefix = "gptoss_gguf_")' in src+ assert (+ 'os.path.join(args.outdir, "gguf")' not in src+ ), "the export is back in the 21GB artifact volume and will fail on space"+++def test_the_off_gpu_walk_names_the_tensors_and_not_only_the_bytes():+ """Driven through the REAL `placement()`, against a stub model.++ `{'cpu': 579133440}` reached hardware twice and nobody could say which+ tensor it was, because the walk used `model.parameters()` and threw the+ names away. 579133440 is exactly 201088 x 2880 -- gpt-oss-20b's vocab by+ its hidden size -- so it was one embedding-shaped tensor all along, and+ that is a filable bug report where a byte count is not.++ A rule fed a hand-written dict would pass either way; this executes the+ producing code, which is the gap that let a missing `torch` import reach+ Kaggle on the Default leg.+ """+ import torch # noqa: PLC0415++ from run_gptoss_t4 import _placement_failures, placement # noqa: PLC0415++ class _Stub:+ def __init__(self):+ self._params = [+ ("model.embed_tokens.weight", torch.zeros(4, 3)),+ ("model.layers.0.mlp.weight", torch.zeros(2, 2)),+ ]++ def named_parameters(self):+ return iter(self._params)++ stub = _Stub()+ record = placement(stub)+ assert record["parameters_by_device"] == {"cpu": 16}+ assert record["off_gpu_parameter_count"] == 2+ # Largest first, so the reader sees the tensor that matters.+ assert [p["name"] for p in record["off_gpu_parameters"]] == [+ "model.embed_tokens.weight",+ "model.layers.0.mlp.weight",+ ]++ failures = _placement_failures(record)+ assert failures, "a wholly-CPU model must fail"+ assert "model.embed_tokens.weight" in failures[0], (+ "the failure message carries only byte counts again, which is the "+ "unactionable red this test exists to prevent"+ )+++def test_a_walk_that_recorded_no_names_still_fails_and_says_so():+ """The refusal branch. An empty name list must not read as "no problem" --+ the byte counts already said there is one."""+ from run_gptoss_t4 import _placement_failures # noqa: PLC0415++ failures = _placement_failures(+ {+ "parameters_by_device": {"cpu": 579133440, "cuda:0": 10461969984},+ "off_gpu_parameters": [],+ "offloaded": False,+ }+ )+ assert len(failures) == 1+ assert "the walk recorded no names" in failures[0]+++def _placement_record(**over):+ """The shape a healthy gpt-oss run produces, measured on+ unsloth-probe-gptoss-names2-ae1968."""+ record = {+ "parameters_by_device": {"cpu": 579133440, "cuda:0": 10461969984},+ "off_gpu_parameters": [+ {"name": "model.embed_tokens.weight", "numel": 579133440, "device": "cpu"}+ ],+ "off_gpu_parameter_count": 1,+ "input_embedding": {+ "module": "model.embed_tokens",+ "weight_name": "model.embed_tokens.weight",+ "device": "cpu",+ "offload_hooks_installed": True,+ },+ "offloaded": False,+ }+ record.update(over)+ return record+++def test_the_deliberate_embedding_offload_is_not_a_failure():+ """Measured, and the assertion was wrong rather than the stack.++ `Unsloth: Offloading embeddings to RAM to save 1.08 GB` is a documented+ optimisation: the input embedding moves to RAM and+ `_install_offload_embedding_hooks` carries ids down and vectors back up. It+ failed this leg twice while training converged, inference was coherent and+ the adapter moved, because the rule read a device count and the count+ cannot tell an optimisation from a spill.+ """+ from run_gptoss_t4 import _placement_failures # noqa: PLC0415+ assert _placement_failures(_placement_record()) == []+++def test_the_excuse_is_the_hook_flag_and_not_the_device():+ """The half that keeps it from being an excuse that can only excuse. An+ embedding on the CPU WITHOUT the hooks is a genuine bug -- the lookup either+ raises or silently synchronises -- and it is indistinguishable from the+ healthy case in `parameters_by_device`."""+ from run_gptoss_t4 import _placement_failures # noqa: PLC0415++ embed = dict(_placement_record()["input_embedding"], offload_hooks_installed = False)+ failures = _placement_failures(_placement_record(input_embedding = embed))+ assert failures and "model.embed_tokens.weight" in failures[0]+++def test_a_second_tensor_off_the_card_is_still_a_failure():+ """The excuse covers exactly one parameter. A real spill that happens to+ include the embedding must not ride in on its coat-tails."""+ from run_gptoss_t4 import _placement_failures # noqa: PLC0415++ record = _placement_record(+ off_gpu_parameters = [+ {"name": "model.embed_tokens.weight", "numel": 579133440, "device": "cpu"},+ {"name": "model.layers.7.mlp.down_proj.weight", "numel": 8294400, "device": "cpu"},+ ]+ )+ failures = _placement_failures(record)+ assert failures+ assert "model.layers.7.mlp.down_proj.weight" in failures[0]+ # The bracketed list is the "what is wrong" half; the embedding is still+ # printed after it as context, which is what makes the verdict readable.+ listed = failures[0].split("[", 1)[1].split("]", 1)[0]+ assert (+ "model.embed_tokens.weight" not in listed+ ), "the list must name what is unexplained, not re-report the tensor that is accounted for"+++def test_the_hook_flag_is_READ_off_the_module_rather_than_assumed():+ """Mutation found this one: hardcoding `offload_hooks_installed = True` in+ `placement()` satisfied every rule above, because they all judge the record+ and none of them produce it. The flag is the entire difference between an+ optimisation and a bug, so it has to come off the module."""+ import torch # noqa: PLC0415++ from run_gptoss_t4 import placement # noqa: PLC0415++ class _Embed(torch.nn.Module):+ def __init__(self):+ super().__init__()+ self.weight = torch.nn.Parameter(torch.zeros(4, 3))++ class _Model(torch.nn.Module):+ def __init__(self):+ super().__init__()+ self.embed_tokens = _Embed()++ def get_input_embeddings(self):+ return self.embed_tokens++ model = _Model()+ assert placement(model)["input_embedding"]["offload_hooks_installed"] is False++ model.embed_tokens._unsloth_offload_hooks_installed = True+ read_back = placement(model)["input_embedding"]+ assert read_back["offload_hooks_installed"] is True+ assert read_back["weight_name"] == "embed_tokens.weight", (+ "the name has to come from the module walk, or it cannot be matched "+ "against the parameter that is off the card"+ )+++def test_the_text_leg_exports_once_per_leg_and_not_once_per_cycle():+ """Measured: the conversion is 310.8s and 312.3s on the two Latest_compile+ cycles and 99.3s and 117.3s on the two vision ones, so the repeat is 47% of+ the longest leg in the suite. The second export is the same base weights+ plus an adapter trained by the same script with the same seed, so it re-runs+ llama.cpp rather than asking a new question; the cycles already prove+ reproducibility on the step tables and the generated text.+ """+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")+ assert 'if getattr(args, "export_gguf", False) and run_index > 0:' in src+ assert '"skipped": "exported on cycle 0' in src+++def test_skipping_every_cycle_is_still_a_failure():+ """The saving must not be able to become missing coverage. A leg that asked+ for an export and produced no file anywhere has to say so, and a per-cycle+ excuse that fires on cycle 0 too would be silent."""+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")+ assert "every cycle skipped the GGUF export" in src+ # The excuse is keyed on a cycle having really exported, not on the flag.+ assert (+ 'exported = [run for run in runs if not (run.get("gguf_export") or {}).get("skipped")]'+ in src+ )+ assert "for run in exported:" in srcdiff --git a/tests/kaggle/test_grpo_nightly.py b/tests/kaggle/test_grpo_nightly.pynew file mode 100644index 00000000000..9df569a8ffa--- /dev/null+++ b/tests/kaggle/test_grpo_nightly.py@@ -0,0 +1,169 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""GRPO runs nightly, and the reason is a measurement rather than a preference.++Over nine T4 sessions the leg hit an intermittent illegal memory access in+vLLM's standby sleep on Turing in FOUR of them. Everything else about it is+sound -- 0.95 utilisation confirmed on both Colab and Kaggle, sleep/wake+surviving three cycles, a non-zero ``reward_std`` on every step once the reward+function stopped saturating -- but a 44% red in front of every PR, for a race no+reader can act on, is exactly how a check gets switched off before the day it is+right. At nightly cadence a clean run still arrives most days and a red one+costs nobody a merge.++So the rules here are about the SHAPE that makes that possible:++* the schedule exists and fires the GRPO leg specifically;+* a leg list REPLACES ``--all-kernels`` rather than filtering after it, because+ the kernel plan and the expected payload count come out of the same call and+ a filter applied afterwards leaves the launcher waiting on payloads nobody+ built;+* the schedule bypasses the sampling gate. A nightly sampled at 15% is a+ weekly, and the difference is invisible until someone goes looking for a+ result that never existed.+"""++from __future__ import annotations++import re+import sys+from pathlib import Path++import yaml++ROOT = Path(__file__).resolve().parents[2]+WORKFLOW = ROOT / ".github" / "workflows" / "kaggle-t4-notebook-ci.yml"+TEXT = WORKFLOW.read_text(encoding = "utf-8")+DOC = yaml.safe_load(TEXT)+TRIGGERS = DOC.get(True) or DOC.get("on")++sys.path.insert(0, str(ROOT / ".github" / "scripts" / "kaggle_t4_ci"))+import legs # noqa: E402+++def test_there_is_a_nightly_schedule():+ assert "schedule" in TRIGGERS, "nothing runs the legs that cannot be per-PR"+ crons = [entry["cron"] for entry in TRIGGERS["schedule"]]+ assert len(crons) == 1, crons+ minute, hour = crons[0].split()[:2]+ assert minute.isdigit() and hour.isdigit(), "the nightly must be a fixed time"+++def test_the_nightly_does_not_pile_onto_the_hour_mark():+ """Every scheduled workflow on GitHub asks for :00, and this one has no+ reason to join that queue."""+ minute = TRIGGERS["schedule"][0]["cron"].split()[0]+ assert minute not in ("0", "00", "30"), f"minute {minute} is the mark everyone else picks"+++def _nightly_legs():+ """The leg names the schedule actually selects, read off the workflow."""+ # Anchored on LEG_LIST, not on the schedule fallback in general. The gate+ # bypass a few lines above reads `github.event_name == 'schedule' && 'true'`+ # and a loose pattern picks THAT up: the first version of this helper+ # reported the nightly leg set as ["true"].+ match = re.search(r"LEG_LIST:.*github\.event_name == 'schedule' && '([a-z_,]+)'", TEXT)+ assert match, "the schedule selects no leg list at all"+ return [name for name in match.group(1).split(",") if name]+++def test_the_schedule_runs_the_grpo_leg():+ assert "grpo" in _nightly_legs(), (+ "the schedule must select the leg; a nightly that runs the wired set "+ "is just another copy of the per-PR run"+ )+++def test_the_schedule_also_runs_the_multi_gpu_leg():+ """multi_gpu is here for a different reason from grpo and the distinction+ is worth keeping: grpo is nightly because it CRASHES 44% of the time,+ multi_gpu because it costs makespan. It passes on hardware -- the ab3 A/B+ has it green in both arms -- and the brief it was built to was multi-GPU+ coverage at no wall-clock cost, which it measurably is not (+172.4s and+ +39.7s over two same-commit pairs, slower in both).++ Nightly is where that coverage is free. Without this the DEVICE_COUNT > 1+ bindings in unsloth's kernels go back to being tested nowhere at all, which+ is the state this leg was written to end."""+ assert "multi_gpu" in _nightly_legs(), (+ "the nightly no longer runs multi_gpu, so unsloth's DEVICE_COUNT > 1 "+ "code path is covered by nothing: every other leg is pinned to one card"+ )+++def test_the_schedule_also_runs_the_latest_compile_leg():+ """Third reason again, and the distinction is the point: grpo is nightly+ because it crashes 44% of sessions, multi_gpu because it costs makespan at+ the margin, latest_compile because it does not FIT.++ Its DONE record is 1323.0s (unsloth-probe-lcleg-tmpdir-ac53ca) and at+ 12.73GB peak it admits no co-tenant, so it wants a whole card for 22+ minutes. The per-PR kernel's only slack is gpu1's 776.3s idle block while+ Studio holds gpu0. 1323 does not go into 776.++ Without this the leg is wired nowhere and gemma-4-E2B-it on the newest+ transformers and trl -- the only thing that caught zoo #1103 -- is tested+ by nothing. That is the state it was built to end, and the reason the leg+ is nightly rather than deleted."""+ assert "latest_compile" in _nightly_legs(), (+ "the nightly no longer runs latest_compile, so nothing anywhere loads "+ "gemma-4-E2B-it on the newest transformers and trl, which is the "+ "pairing that found unsloth-zoo #1103"+ )+++def test_every_leg_the_nightly_names_exists():+ """A typo here produces a build that selects nothing and a run that proves+ nothing, with no error anywhere."""+ named = _nightly_legs()+ assert named, "no scheduled leg name found at all"+ for name in named:+ assert name in legs.LEGS, f"the nightly names {name!r}, which is not a leg"+++def test_the_nightly_set_fits_in_one_kernel():+ """`--legs` builds ONE kernel, and MAX_LEGS_PER_KERNEL is what the driver's+ scheduling was measured against. A list longer than that silently packs a+ kernel nobody has run."""+ assert len(_nightly_legs()) <= legs.MAX_LEGS_PER_KERNEL+++def test_no_nightly_leg_is_ALSO_in_the_per_pr_set():+ """The whole point, and it applies to each of them. If grpo were wired into+ KERNELS the 44% crash rate would be back in front of every PR; if multi_gpu+ were, the makespan it was moved here to avoid would be back too. Either way+ the nightly becomes a second copy of the per-PR run."""+ wired = {name for kernel in legs.KERNELS for name in kernel}+ both = sorted(set(_nightly_legs()) & wired)+ assert not both, (+ f"{both} run nightly AND per-PR, so the nightly is pointless and every "+ f"PR carries whatever these were moved off the critical path to avoid"+ )+++def test_a_leg_list_replaces_all_kernels_rather_than_filtering_after_it():+ """``--all-kernels`` derives BOTH the kernel plan and the payload count the+ launcher waits on. A filter applied afterwards leaves it expecting payloads+ that were never built, which times out rather than failing."""+ assert 'KERNEL_SELECT="--legs $LEG_LIST"' in TEXT+ assert 'KERNEL_SELECT="--all-kernels"' in TEXT+ assert "$KERNEL_SELECT \\" in TEXT+ assert (+ "--all-kernels \\" not in TEXT+ ), "--all-kernels is still hardcoded, so the override cannot take effect"+++def test_the_schedule_bypasses_the_sampling_gate():+ """A nightly sampled at 15% is a weekly, and the difference is invisible+ until someone goes looking for a result that never existed."""+ assert "github.event_name == 'schedule' && 'true'" in TEXT, (+ "the schedule does not force the gate, so most nights it will draw a "+ "stand-down and report nothing"+ )+++def test_the_leg_list_default_survives_a_schedule_event():+ """``inputs`` is null on a schedule, so an input default cannot supply the+ value; it has to come from the fallback."""+ assert "inputs.legs || (github.event_name == 'schedule'" in TEXTdiff --git a/tests/kaggle/test_harness_dependencies.py b/tests/kaggle/test_harness_dependencies.pynew file mode 100644index 00000000000..a99c3dba65f--- /dev/null+++ b/tests/kaggle/test_harness_dependencies.py@@ -0,0 +1,169 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""The CI step that runs these tests must be able to import what they import.++The failure this exists against is quiet by construction. `Test the harness`+runs `pytest tests/kaggle -q` against a hand-written pip line, and a module+missing from that line does not stop the step: pytest reports the collection or+call error, the step goes red with everything else green, and the two tests that+could not import are indistinguishable in a summary from tests that never+existed. Measured on run 32925226213, which ran **803 of 805** because+`datasets` was not installed -- and the two it lost are the pair proving+`with_transform` keeps PIL images while `Dataset.from_list` corrupts them, which+is the finding the vision leg was built on.++`pillow` and `pyyaml` were in the same position and passing only because+`transformers` and friends happened to pull them. A transitive dependency is not+a dependency; it is a coincidence that holds until an upstream drops it.++So the rule is an agreement between two things that are edited months apart: the+modules `tests/kaggle/test_*.py` import without a `pytest.importorskip` guard,+and the distributions the workflow installs.++Scope is deliberately the TEST modules, not the payloads under `t4_smoke/` and+`studio_gpu/`. Those import trl, unsloth, unsloth_zoo and vllm, and they do it+lazily inside functions because they run on a Kaggle GPU session rather than on+this runner. Demanding them here would install a CUDA stack on an+`ubuntu-latest` box to run a pure-AST test suite.+"""++from __future__ import annotations++import ast+import re+import sys+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+TESTS = ROOT / "tests" / "kaggle"+WORKFLOW = ROOT / ".github" / "workflows" / "kaggle-t4-notebook-ci.yml"++# Import name -> the distribution that provides it, for the ones that differ.+# Anything not listed is assumed to install under its own name.+DISTRIBUTION = {+ "PIL": "pillow",+ "yaml": "pyyaml",+}++# Directories whose module names are importable because the suite puts them on+# sys.path, so they are not third-party however they are spelled.+LOCAL_DIRS = (+ TESTS,+ TESTS / "t4_smoke",+ TESTS / "studio_gpu",+ ROOT / ".github" / "scripts",+ ROOT / ".github" / "scripts" / "kaggle_t4_ci",+ ROOT / ".github" / "scripts" / "kaggle_studio_ci",+)+++def _local_names() -> set[str]:+ names = set()+ for directory in LOCAL_DIRS:+ names.add(directory.name)+ for path in directory.glob("*.py"):+ names.add(path.stem)+ return names+++def _imported_third_party() -> dict[str, set[str]]:+ """Every third-party module the test modules import, and where.++ Any depth, not just module level: a `from datasets import Dataset` inside a+ test function fails when that test RUNS, which is exactly the case run+ 32925226213 hit, and a module-level-only scan would have missed it.+ """+ local = _local_names()+ found: dict[str, set[str]] = {}+ for path in sorted(TESTS.glob("test_*.py")):+ tree = ast.parse(path.read_text(encoding = "utf-8"))+ # `pytest.importorskip("x")` is the suite's own way of saying a module+ # is optional, so it is not a claim on the install line.+ guarded = {+ node.args[0].value.split(".")[0]+ for node in ast.walk(tree)+ if isinstance(node, ast.Call)+ and getattr(node.func, "attr", "") == "importorskip"+ and node.args+ and isinstance(node.args[0], ast.Constant)+ and isinstance(node.args[0].value, str)+ }+ for node in ast.walk(tree):+ names: list[str] = []+ if isinstance(node, ast.Import):+ names = [alias.name.split(".")[0] for alias in node.names]+ elif isinstance(node, ast.ImportFrom) and node.module and node.level == 0:+ names = [node.module.split(".")[0]]+ for name in names:+ if name in sys.stdlib_module_names or name in local or name in guarded:+ continue+ found.setdefault(name, set()).add(path.name)+ return found+++def _install_line() -> str:+ text = WORKFLOW.read_text(encoding = "utf-8")+ step = text.index("- name: Test the harness")+ body = text[step : text.index("\n - name:", step + 1)]+ matches = re.findall(r"pip install[^\n]*", body)+ assert matches, "the Test the harness step no longer installs anything"+ return " ".join(matches)+++def test_every_unguarded_import_is_installed_by_the_workflow():+ """The whole point. A module missing here costs tests silently."""+ line = _install_line()+ missing = {+ module: sorted(files)+ for module, files in _imported_third_party().items()+ if DISTRIBUTION.get(module, module) not in line+ }+ assert not missing, (+ "these modules are imported by tests/kaggle/test_*.py without a "+ "pytest.importorskip guard and are not installed by the Test the "+ f"harness step, so those tests will error rather than run: {missing}"+ )+++def test_the_import_scan_finds_something_at_all():+ """A scan that silently matched nothing would satisfy the rule above+ forever. Two modules the suite genuinely imports are named here, so a+ refactor that breaks the walk fails rather than passes."""+ found = _imported_third_party()+ assert "datasets" in found, "the walk no longer sees the vision dataset imports"+ assert "torch" in found+ assert found["datasets"], "no file recorded for datasets"+++def test_the_scan_looks_inside_functions_and_not_only_at_module_level():+ """The measured failure was a function-level import. `test_vision_run.py`+ imports `datasets` inside two test bodies and nowhere else, so a+ module-level-only scan reports a clean tree and the step still loses two+ tests."""+ source = (TESTS / "test_vision_run.py").read_text(encoding = "utf-8")+ tree = ast.parse(source)+ module_level = {+ alias.name.split(".")[0]+ for node in tree.body+ if isinstance(node, ast.Import)+ for alias in node.names+ } | {+ node.module.split(".")[0]+ for node in tree.body+ if isinstance(node, ast.ImportFrom) and node.module+ }+ assert "datasets" not in module_level, (+ "datasets is now imported at module level in test_vision_run.py, which "+ "makes this test tautological; point it at another function-level "+ "import instead of deleting it"+ )+ assert "datasets" in _imported_third_party()+++def test_a_guarded_import_is_not_treated_as_a_requirement():+ """`pytest.importorskip` is how this suite says a module is optional, and+ demanding those on the runner would put a CUDA stack on an ubuntu box."""+ line = _install_line()+ assert "vllm" not in line+ assert "unsloth" not in linediff --git a/tests/kaggle/test_kernel_provenance.py b/tests/kaggle/test_kernel_provenance.pynew file mode 100644index 00000000000..9d7a4f01f29--- /dev/null+++ b/tests/kaggle/test_kernel_provenance.py@@ -0,0 +1,176 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""Fast-kernel provenance, and the three ways this check goes vacuous.++Every rule here is calibrated against what was MEASURED on a real 2xT4+(`unsloth-probe-vision-recon-c76ea3`), not against what the brief assumed:++* `fla` resolves to `unsloth_zoo/_vendored/fla`, version 0.5.1, and only AFTER+ the model load;+* attention resolves to `sdpa`, and `flash_attn` is not importable at all;+* `causal_conv1d` and `mamba_ssm` are NOT installed on this path, before or+ after the load.++The last one is why two obvious assertions are absent: asserting them present+would be red on correct behaviour.+"""++from __future__ import annotations++import sys+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "t4_smoke"+sys.path.insert(0, str(PAYLOAD))++from kernel_provenance import vision_kernel_failures # noqa: E402++VENDORED = {+ "importable": True,+ "file": "/usr/local/lib/python3.12/dist-packages/unsloth_zoo/_vendored/fla/__init__.py",+ "version": "0.5.1",+ "vendored": True,+}+++def test_the_measured_configuration_passes():+ """The exact shape the recon probe recorded. If this ever fails, the rule+ has drifted away from the hardware rather than the hardware from the rule."""+ kernels = {+ "fla": VENDORED,+ "causal_conv1d": {"importable": False, "error": "ModuleNotFoundError"},+ "mamba_ssm": {"importable": False, "error": "ModuleNotFoundError"},+ "flash_attn": {"importable": False, "error": "ModuleNotFoundError"},+ }+ assert vision_kernel_failures(kernels, {"config": "sdpa"}, capability = "7.5") == []+++def test_a_pip_installed_fla_is_not_the_vendored_one():+ """ "importable" is the tempting assertion and it is the wrong one: it passes+ on a copy that is not what ships."""+ kernels = {+ "fla": {"importable": True, "file": "/site-packages/fla/__init__.py", "vendored": False}+ }+ broken = vision_kernel_failures(kernels, {"config": "sdpa"}, capability = "7.5")+ assert broken and "not the vendored copy" in broken[0]+++def test_fla_missing_is_a_failure():+ kernels = {"fla": {"importable": False, "error": "ModuleNotFoundError: fla"}}+ broken = vision_kernel_failures(kernels, {"config": "sdpa"}, capability = "7.5")+ assert broken and "did not import after the model load" in broken[0]+++def test_flash_attention_2_on_turing_is_a_failure():+ """The regression that matters. FA2 cannot execute on sm_75, so a stack that+ selects it fails at the first forward, and a leg that did not check would+ report that as an unexplained crash."""+ broken = vision_kernel_failures(+ {"fla": VENDORED}, {"config": "flash_attention_2"}, capability = "7.5"+ )+ assert broken and "cannot run it" in broken[0]+++def test_flash_attention_2_is_not_flagged_off_turing():+ """The same choice is correct on Ampere. A rule that fired everywhere would+ be wrong rather than strict."""+ assert (+ vision_kernel_failures({"fla": VENDORED}, {"config": "flash_attention_2"}, capability = "8.6")+ == []+ )+++def test_a_missing_attention_record_is_a_failure_not_a_silence():+ broken = vision_kernel_failures({"fla": VENDORED}, {}, capability = "7.5")+ assert broken and "no attention implementation was recorded" in broken[0]+++def test_causal_conv1d_absence_is_reported_and_not_asserted():+ """Measured absent on this path. The wheel-first machinery in ssm_runtime.py+ belongs to Studio's training worker, which the notebook path never calls, so+ an assertion here would go red on correct behaviour."""+ kernels = {+ "fla": VENDORED,+ "causal_conv1d": {"importable": False, "error": "ModuleNotFoundError"},+ "mamba_ssm": {"importable": False, "error": "ModuleNotFoundError"},+ }+ assert vision_kernel_failures(kernels, {"config": "sdpa"}, capability = "7.5") == []+++def test_no_provenance_at_all_is_a_failure():+ assert vision_kernel_failures(None, {"config": "sdpa"}, capability = "7.5")+++def test_both_capability_spellings_reach_the_turing_rule():+ """The bug this catches was mine, and it was live for a few minutes.++ `environment_fingerprint()` records `"sm_75"`, while the recon probe and+ `torch.cuda.get_device_capability` give `"7.5"`. A `startswith("7.")` check+ against `"sm_75"` matches nothing, so the FA2 rule would never fire and the+ leg would report a clean pass while checking nothing at all.+ """+ for spelling in ("7.5", "sm_75", "75"):+ broken = vision_kernel_failures(+ {"fla": VENDORED}, {"config": "flash_attention_2"}, capability = spelling+ )+ assert broken, f"the Turing rule never fired for capability={spelling!r}"++ for spelling in ("8.6", "sm_86", "86", ""):+ assert (+ vision_kernel_failures(+ {"fla": VENDORED}, {"config": "flash_attention_2"}, capability = spelling+ )+ == []+ ), f"the Turing rule fired for capability={spelling!r}"+++def test_the_payload_passes_the_capability_the_fingerprint_records():+ """Asserted from the source: the wiring is where the two spellings meet."""+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")+ assert 'capability = str(env.get("gpu_capability"' in src+++def test_the_payload_never_calls_a_tokenizer_positionally():+ """A vision model's tokenizer IS a processor.++ `ProcessorMixin.__call__` is `(self, images=None, text=None, videos=None,+ ...)`, so a positional list of prompts is taken as IMAGES and transformers+ tries to fetch each string as an image URL. That is not hypothetical: it+ killed the Latest_compile leg on gemma-4-E2B-it after the model had already+ loaded and trained, on kernel unsloth-probe-latestcompile-r2-62b54d.++ `text` is also the first parameter of a plain tokenizer, so the keyword is+ correct everywhere and this is not a vision special case.+ """+ import re++ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")+ offenders = []+ for match in re.finditer(r"tokenizer\(\s*(?!text\s*=)(?!\))([^)\n]*)", src):+ arg = match.group(1).strip()+ # Keyword-only calls are fine; a bare `tokenizer(` opening a keyword+ # list is what the negative lookahead already allowed through.+ if arg and not arg.split(",")[0].strip().endswith("=") and "=" not in arg.split(",")[0]:+ offenders.append(match.group(0))+ assert offenders == [], f"positional tokenizer calls: {offenders}"+++def test_prompt_token_lengths_index_past_the_batch_dimension():+ """The missing `[0]` made the whole padding check vacuous.++ A processor returns `input_ids` with a BATCH dimension, so `len(...)` on it+ is the number of sequences -- 1 -- for every prompt. Measured on kernel+ unsloth-probe-latestcompile-r3-cb1125, where gemma-4 reported+ `[1, 1, 1, 1, 1, 1, 1, 1]` and the run's own vacuity guard caught it:++ every batched prompt tokenised to the same length, so nothing was ever+ padded and the left-padding check proved nothing++ A plain tokenizer given one string returns a flat list, which is why this+ read correctly on every text model and broke on the first vision one.+ """+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")+ assert 'len(tokenizer(text = [p])["input_ids"][0]) for p in prompts' in src+ assert 'len(tokenizer(text = p)["input_ids"]) for p in prompts' not in srcdiff --git a/tests/kaggle/test_known_upstream_breakage.py b/tests/kaggle/test_known_upstream_breakage.pynew file mode 100644index 00000000000..8500e5588df--- /dev/null+++ b/tests/kaggle/test_known_upstream_breakage.py@@ -0,0 +1,104 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""A confirmed upstream bug must not block a leg, and must not outlive itself.++Batched greedy generation on `unsloth/gemma-4-E2B-it` and `unsloth/Qwen3.5-2B`+disagrees at sizes 2, 4 and 8 with real left padding and demonstrably distinct+prompt lengths, on both repeats, on a current stack. It is filed as unsloth+#9708 and it is not this repo's to fix. Leaving it as a hard failure means+those two legs put a red in front of every PR for a bug no reader can act on,+which is how a check gets switched off.++So the entry is a STRICT expectation rather than a mute, and the difference is+the whole point of this file. A model listed as broken that starts AGREEING+FAILS, with a message saying to delete the entry. An excuse that can only ever+excuse is indistinguishable from deleted coverage, and it outlives the bug it+was written for by years.++Everything else stays live for those models. The padding side, the distinct+lengths, the empty-output check and the batch-size floor are what make the+disagreement a real finding rather than an unpadded batch, and none of them is+touched by the entry.+"""++from __future__ import annotations++import sys+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "t4_smoke"+sys.path.insert(0, str(PAYLOAD))++from run_t4_smoke import ( # noqa: E402+ KNOWN_BATCHED_GENERATION_BREAKAGE,+ batched_generation_failures,+)+++def _record(**over):+ base = {+ "prompt_token_lengths": [13, 13, 14, 15, 13, 13, 15, 16],+ "distinct_lengths": 4,+ "padding_side_observed": "left",+ "padding_side_after": "left",+ "singles": ["x"] * 8,+ "batched": {"2": ["y"] * 8, "4": ["y"] * 8, "8": ["y"] * 8},+ "agrees": {"2": False, "4": False, "8": False},+ "empty_outputs": [],+ }+ base.update(over)+ return base+++def test_an_unlisted_model_still_fails_on_disagreement():+ """The rule this whole check exists for is unchanged for everything else."""+ broken = batched_generation_failures(_record(), "unsloth/Qwen3-0.6B")+ assert len(broken) == 3+ assert all("did not reproduce" in f for f in broken)+++def test_a_listed_model_does_not_fail_on_the_known_disagreement():+ assert "unsloth/gemma-4-E2B-it" in KNOWN_BATCHED_GENERATION_BREAKAGE+ assert batched_generation_failures(_record(), "unsloth/gemma-4-E2B-it") == []+++def test_a_listed_model_that_starts_AGREEING_fails():+ """The strict half, and the reason this is not a mute. The day #9708 is+ fixed, CI says so instead of carrying a stale excuse."""+ agreeing = _record(agrees = {"2": True, "4": True, "8": True})+ broken = batched_generation_failures(agreeing, "unsloth/Qwen3.5-2B")+ assert broken, "a fixed upstream bug must turn the leg red"+ assert "delete the entry" in broken[0]+++def test_a_listed_model_still_fails_every_other_rule():+ """The entry excuses ONE claim. A right-padded batch, an unpadded batch or+ an empty output is still a failure, and those are exactly what make the+ disagreement meaningful rather than an artefact."""+ for over, expect in (+ ({"padding_side_after": "right"}, "padding_side_after"),+ ({"distinct_lengths": 1}, "nothing was ever padded"),+ ({"empty_outputs": [3]}, "generated nothing at all"),+ ({"singles": ["x"] * 2}, "largest batch was never actually formed"),+ ):+ broken = batched_generation_failures(_record(**over), "unsloth/Qwen3.5-2B")+ assert any(+ expect in f for f in broken+ ), f"{over} was excused along with the known disagreement"+++def test_every_entry_names_the_issue_it_is_waiting_on():+ """An excuse with no issue number is a mute with better manners."""+ for model, reference in KNOWN_BATCHED_GENERATION_BREAKAGE.items():+ assert reference and "#" in reference, f"{model} names no issue"+++def test_the_list_is_short_and_explicit():+ """A growing list is the signal that this mechanism is being used to make+ reds go away rather than to carry one filed bug."""+ assert len(KNOWN_BATCHED_GENERATION_BREAKAGE) <= 3, (+ "if this list is growing, the mechanism is being used to silence "+ "failures rather than to hold one filed upstream bug"+ )diff --git a/tests/kaggle/test_multi_gpu_leg.py b/tests/kaggle/test_multi_gpu_leg.pynew file mode 100644index 00000000000..371792856fa--- /dev/null+++ b/tests/kaggle/test_multi_gpu_leg.py@@ -0,0 +1,505 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""The multi-card leg: a branch no pinned payload can reach.++``unsloth/kernels/utils.py:170``::++ if DEVICE_COUNT > 1:+ torch_gpu_device = torch.cuda.device # a real device switch+ else:+ def torch_gpu_device(device): return nullcontext()++``build_kernel.py`` pins every ordinary payload with ``CUDA_VISIBLE_DEVICES``,+so **every unsloth kernel this CI has ever run took the nullcontext branch** --+and so did the ``DEVICE_COUNT``-sized ``CUDA_STREAMS`` / ``WEIGHT_BUFFERS`` /+``ABSMAX_BUFFERS`` arrays, the per-device rotary caches in+``unsloth/models/llama.py:1838``, and the ``temp_mlp`` device tuples at+``llama.py:1300``.++The leg is free because the divergence is triggered by VISIBILITY, not by using+both cards: the model still fits on one, so it co-tenants and the driver+reserves its 0.7 GB on each card rather than a whole one.++**What these rules deliberately do NOT assert** is that the parameters end up+spread across both cards. Whether accelerate shards or pins to ``cuda:0`` is+the open question the leg exists to answer, and a rule written before the+answer is a rule written to match whatever happens.+"""++from __future__ import annotations++import importlib.util+import sys+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+SMOKE_DIR = ROOT / "tests" / "kaggle" / "t4_smoke"+sys.path.insert(0, str(ROOT / ".github" / "scripts"))++from kaggle_t4_ci import build_kernel, legs # noqa: E402+++def _payload():+ spec = importlib.util.spec_from_file_location(+ "_t4_smoke_multi_gpu", SMOKE_DIR / "run_t4_smoke.py"+ )+ module = importlib.util.module_from_spec(spec)+ sys.path.insert(0, str(SMOKE_DIR))+ spec.loader.exec_module(module)+ return module+++GOOD = {+ "device_count": 2,+ "module_device_count": 2,+ "torch_gpu_device_is_real_switch": True,+ "torch_gpu_device_repr": "<class 'torch.cuda.device'>",+ "cuda_streams_len": 2,+ "weight_buffers_len": 2,+ "absmax_buffers_len": 2,+ "rotary_cache_slots": 2,+ "parameters_by_device": {"cuda:0": 596049920},+ "cuda_devices_holding_parameters": ["cuda:0"],+}+++def test_the_happy_reading_passes():+ assert _payload().multi_gpu_failures(GOOD, expected_cards = 2) == []+++def test_the_nullcontext_shim_is_the_failure_this_leg_exists_for():+ """The single most important rule here. A leg that ran on one card, or+ whose unsloth was imported before the cards were visible, gets the shim --+ which performs NO device switch -- and would otherwise report a pass for+ coverage a pinned leg already has."""+ facts = dict(+ GOOD,+ torch_gpu_device_is_real_switch = False,+ torch_gpu_device_repr = "<function torch_gpu_device at 0x7f>",+ )+ broken = _payload().multi_gpu_failures(facts, expected_cards = 2)+ assert broken and "nullcontext" in broken[0]+++def test_one_visible_card_fails_and_says_only_that():+ """A pinned run measures the wrong machine, so every rule after the count+ is noise. Reporting eight failures about arrays that are correctly sized+ for the one card the leg was given sends the reader after the wrong bug."""+ facts = dict(+ GOOD,+ device_count = 1,+ module_device_count = 1,+ torch_gpu_device_is_real_switch = False,+ cuda_streams_len = 1,+ weight_buffers_len = 1,+ absmax_buffers_len = 1,+ rotary_cache_slots = 1,+ )+ broken = _payload().multi_gpu_failures(facts, expected_cards = 2)+ assert len(broken) == 1+ assert "the driver pinned it" in broken[0]+++def test_an_import_before_the_cards_were_visible_is_caught_separately():+ """torch can see two cards while unsloth.kernels.utils was imported when+ only one was visible -- the bindings are made once, at import. The two are+ different failures with different fixes, so they are reported separately."""+ facts = dict(GOOD, module_device_count = 1)+ broken = _payload().multi_gpu_failures(facts, expected_cards = 2)+ assert broken and "imported before the cards were visible" in broken[0]+++def test_short_stream_and_buffer_arrays_each_fail():+ payload = _payload()+ for key in ("cuda_streams_len", "weight_buffers_len", "absmax_buffers_len"):+ broken = payload.multi_gpu_failures(dict(GOOD, **{key: 1}), expected_cards = 2)+ assert broken, key+ assert "no stream or buffer of its own" in broken[0], key+ # None is not "fine": a missing array means the attribute is gone.+ assert payload.multi_gpu_failures(dict(GOOD, **{key: None}), expected_cards = 2)+++def test_missing_facts_are_a_failure_rather_than_a_silence():+ """A cycle that never ran the reading reports null, and null must not read+ as "nothing wrong". This is the shape that let a payload be carried and+ never executed for two rounds."""+ payload = _payload()+ assert payload.multi_gpu_failures(None, expected_cards = 2)+ assert payload.multi_gpu_failures({}, expected_cards = 2)+ broken = payload.multi_gpu_failures({"error": "ImportError: no unsloth"}, expected_cards = 2)+ assert broken and "could not be read" in broken[0]+++def test_a_model_entirely_off_the_gpu_fails():+ facts = dict(GOOD, parameters_by_device = {"cpu": 596049920}, cuda_devices_holding_parameters = [])+ assert _payload().multi_gpu_failures(facts, expected_cards = 2)+++def test_the_spread_across_cards_is_RECORDED_and_not_required():+ """The open question, and the rule must not pre-empt it.++ Every two-card measurement this repo has is a LOAD and not a train+ (unsloth-probe-vision-recon-c76ea3 saw a model split 897.7/1017.1 MB and+ never called a trainer). So a single-card placement passes, a spread+ passes, and the report carries which happened. When a session answers it,+ tighten this deliberately rather than discovering the rule was already+ asserting an answer nobody had.+ """+ payload = _payload()+ one_card = dict(GOOD, cuda_devices_holding_parameters = ["cuda:0"])+ spread = dict(+ GOOD,+ parameters_by_device = {"cuda:0": 300000000, "cuda:1": 296049920},+ cuda_devices_holding_parameters = ["cuda:0", "cuda:1"],+ )+ assert payload.multi_gpu_failures(one_card, expected_cards = 2) == []+ assert payload.multi_gpu_failures(spread, expected_cards = 2) == []+++def test_the_reading_is_taken_from_the_module_and_not_recomputed():+ """`torch.cuda.device_count()` on both sides of the comparison is how a+ rule ends up unable to fail: the binding is made once at import, so asking+ torch again answers a different question and answers it agreeably."""+ import ast++ source = (SMOKE_DIR / "run_t4_smoke.py").read_text(encoding = "utf-8")+ func = next(+ n+ for n in ast.walk(ast.parse(source))+ if isinstance(n, ast.FunctionDef) and n.name == "multi_gpu_facts"+ )+ body = ast.unparse(func)+ assert "from unsloth.kernels import utils" in body+ assert "binding is torch.cuda.device" in body+ assert "DEVICE_COUNT" in body+++# ------------------------------------------------------------ the leg itself+++def test_the_leg_asks_for_two_cards_and_the_BUILT_payload_enforces_it():+ """Asserted through the generated notebook, not off the dataclass. A field+ nothing emits reads like coverage and does nothing -- which is exactly how+ a payload was carried and never run for two rounds."""+ leg = legs.LEGS["multi_gpu"]+ assert leg.all_cards is True+ notebook = build_kernel.build_payload_notebook(+ SMOKE_DIR, leg, unsloth_ref = "main", zoo_ref = "main"+ )+ source = "".join("".join(c["source"]) for c in notebook["cells"])+ assert "device_count() == 2" in source, (+ "the built payload does not require two visible GPUs, so a run the "+ "driver pinned would measure the single-card branch and pass"+ )+ assert "--require-multi-gpu" in source+ assert "--expected-cards" in source+++def test_every_OTHER_leg_still_requires_exactly_one():+ """The pin check is what catches a driver that failed to pin, and widening+ it for this leg must not widen it for the rest."""+ for name, leg in legs.LEGS.items():+ if name == "multi_gpu":+ continue+ notebook = build_kernel.build_payload_notebook(+ SMOKE_DIR, leg, unsloth_ref = "main", zoo_ref = "main"+ )+ source = "".join("".join(c["source"]) for c in notebook["cells"])+ assert "device_count() == 1" in source, name+++def test_the_leg_is_small_enough_to_co_tenant():+ """The whole cost argument. At a whole card this leg would block gptoss+ (12.78 of the 13.0 budget) for its entire life and the kernel would get+ longer, which is the one thing this was built not to do."""+ leg = legs.LEGS["multi_gpu"]+ budget = 13.0 # CARD_VRAM_BUDGET_GB in build_kernel.py++ # Expressed against the budget rather than a round number. The first+ # version of this rule asserted `vram_gb <= 1.0`, which was a figure with no+ # reasoning behind it, and it went red the moment the declaration was+ # corrected from a guessed 0.7 to the measured 1.2 -- a guard failing on a+ # measurement replacing a guess is the guard being wrong.+ #+ # What has to hold is that this leg can still SHARE with any ordinary leg,+ # since running on both cards while excluding everything from both is the+ # opposite of free.+ others = [+ other.vram_gb+ for name, other in legs.LEGS.items()+ if name != "multi_gpu" and other.vram_gb < budget / 2+ ]+ assert others+ assert leg.vram_gb + max(others) <= budget, (+ f"at {leg.vram_gb} GB on EVERY card this leg cannot share with a "+ f"{max(others)} GB leg, so it takes both cards outright"+ )+ assert leg.vram_gb + legs.LEGS["gptoss"].vram_gb > budget, (+ "gptoss must still be unable to share a card with this leg; if that "+ "stopped being true the co-tenancy argument changed"+ )+++def test_the_declaration_is_measured_and_not_copied_from_a_sibling():+ """`vram_gb` was 0.7 for one commit, copied from the other Qwen legs. It is+ wrong for this leg specifically -- it holds a CUDA context on BOTH cards --+ and the repo-wide check could not say so, because it compares against+ measured_vram.json and this leg was simply absent from it. A guard that+ passes by finding nothing is the shape this directory keeps being caught+ by."""+ import json++ measured = json.loads((SMOKE_DIR / "measured_vram.json").read_text(encoding = "utf-8"))[+ "peak_reserved_gb"+ ]+ assert (+ "multi_gpu" in measured+ ), "the leg is not in the measured file, so the declaration check passes on it vacuously"+ assert legs.LEGS["multi_gpu"].vram_gb >= measured["multi_gpu"]+++def test_it_does_not_export_a_gguf_and_the_reason_is_recorded():+ """The bundle install_llama_cpp fetches for the notebook legs is the CPU+ one -- on unsloth-probe-full-concurrent-417238 this model's llama-bench+ reports `backend CPU` -- so a two-card tensor-split assertion here could+ not fail. Adding the export anyway would cost ~40s of a thin margin and+ buy a claim four other legs already make."""+ leg = legs.LEGS["multi_gpu"]+ assert "--export-gguf" not in leg.args+ # The reason travels with the decision. A leg that simply lacks a flag+ # invites someone to add it back in the hour they notice.+ source = (ROOT / ".github" / "scripts" / "kaggle_t4_ci" / "legs.py").read_text(encoding = "utf-8")+ entry = source.split('"multi_gpu": Leg(')[1].split('),\n "')[0]+ assert "backend CPU" in entry or "CPU bundle" in entry+++# --------------------------------------------------------- driver scheduling+++def _driver_source() -> str:+ payloads = {}+ for name in ("default", "gptoss", "multi_gpu"):+ leg = legs.LEGS[name]+ payloads[f"t4_{leg.name}.ipynb"] = build_kernel.build_payload_notebook(+ SMOKE_DIR, leg, unsloth_ref = "main", zoo_ref = "main"+ )+ driver = build_kernel.build_driver(+ payloads,+ per_run_timeout = 3600,+ vram_source = {+ f"t4_{legs.LEGS[n].name}.ipynb": legs.LEGS[n]+ for n in ("default", "gptoss", "multi_gpu")+ },+ all_card = ("t4_Multi_GPU.ipynb",),+ )+ return "".join("".join(c["source"]) for c in driver["cells"])+++def test_the_all_card_payload_is_kept_out_of_the_card_queue():+ source = _driver_source()+ assert "ALL_CARD = (" in source+ assert "t4_Multi_GPU.ipynb" in source.split("ALL_CARD = ")[1][:200]+ assert "_queue = [(i, n) for i, n in enumerate(ORDER) if n not in ALL_CARD]" in source+++def test_it_runs_UNPINNED_or_it_measures_the_single_card_branch():+ """`run_one(name, None, idx)`. An index here sets CUDA_VISIBLE_DEVICES and+ the leg would take the nullcontext branch under a multi-GPU name."""+ source = _driver_source()+ lane = source.split("def _all_card_lane")[1].split("threads = []")[0]+ assert "run_one(name, None, idx)" in lane+ assert "run_one(name, 0" not in lane+++def test_the_reservation_is_all_or_nothing():+ """A partial reservation holds a seat on card 0 while waiting for card 1 --+ capacity nothing is using, and a deadlock against a big leg waiting for+ card 0. The refusal path must not have mutated anything."""+ source = _driver_source()+ body = source.split("def _admit_all(name):")[1].split("def _release_all")[0]+ check, commit = body.split("for g in range(N_GPU):")[1:3]+ # Every rejection happens BEFORE any card is charged.+ assert "return False" in check+ assert "return False" not in commit+ assert "card_load[g] += want" in commit+ assert "card_load[g] += want" not in check+++def test_the_contention_is_global_and_the_vram_is_per_card():+ """The two ledgers track different things, and charging the wrong one to a+ card has now cost twice.++ `card_load` is memory. An all-card leg really does hold a CUDA context on+ each card -- 1.2 GB measured on unsloth-probe-multigpu-r2-a280e2 -- so it is+ charged to every card.++ `card_count` is a proxy for 4-vCPU CONTENTION; `MAX_LEGS_PER_CARD`'s own+ comment says the legs "contend for CORES long before they contend for+ memory". An unpinned process contends for those cores without occupying a+ card, so it counts against the TOTAL bound and against no card in+ particular.++ Charged per card it was wrong in both available ways. On every card, the+ driver simulation showed no card holding two legs at once. On ONE card, it+ decided a placement it had nothing to do with and cost 188.7s on hardware:+ in unsloth-probe-ab-with-multigpu-6169ca Studio was refused gpu0 (canary+ plus this leg's count = the cap) and took gpu1, where the 1707s vision leg+ had ~1080s left, so gpu1 carried both long payloads and gpu0 idled 622.6s.+ The same kernel without the leg (unsloth-probe-ab-baseline-5leg-20db9c) ran+ 1936.4s with 10.0s and 3.0s of idle.+ """+ source = _driver_source()+ body = source.split("def _admit_all(name):")[1].split("def _release_all")[0]+ assert "unpinned_count[0] += 1" in body+ assert "card_count[" not in body.split("card_load[g] += want")[1], (+ "the all-card leg is charging a card slot again, which is what put the "+ "two longest payloads on one card and idled the other for 622.6s"+ )+ # The global bound has to be enforced on BOTH paths. In _admit_all only, the+ # card queue fills up beside the unpinned leg and puts one more install than+ # the bound allows onto four cores.+ pinned = source.split("def _admit(gpu_index, name):")[1].split("def _release")[0]+ for where, text in (("_admit", pinned), ("_admit_all", body)):+ assert "unpinned_count[0] >= N_GPU * MAX_LEGS_PER_CARD" in text, where+ release = source.split("def _release_all(name):")[1].split("\n\n")[0]+ assert "unpinned_count[0] -= 1" in release+++def test_the_all_card_lane_is_joined_with_the_card_workers():+ """It runs BESIDE the training legs, so anything concluding the cards are+ free has to wait for it too."""+ source = _driver_source()+ assert "for t in all_card_threads:" in source+ assert source.index("for t in all_card_threads:") < source.index(+ "if AFTER_GPU and not AFTER_GPU_CONCURRENT:"+ )+++def test_a_single_leg_dispatch_still_stands_down_on_one_card():+ """`expected_gpus = min(len(payloads), SESSION_GPUS)` derives 1 for a+ one-leg kernel, which is right for every leg but this one.++ An all-card leg on a one-card allocation is an INFRASTRUCTURE fact, and+ without this it arrives as a payload failure on `device_count() == 2` --+ the exact confusion the shortfall guard's own comment says it exists to+ prevent.+ """+ notebooks = build_kernel.build_kernel(+ SMOKE_DIR,+ ("multi_gpu",),+ unsloth_ref = "main",+ zoo_ref = "main",+ extra_args = (),+ per_run_timeout = 3600,+ )+ source = "".join("".join(c["source"]) for c in notebooks["cells"])+ assert "EXPECTED_GPUS = 2" in source, (+ "a one-leg multi_gpu dispatch accepts a single card, so the leg would "+ "fail inside the payload instead of standing the kernel down"+ )+ # And an ordinary one-leg dispatch is unchanged.+ other = build_kernel.build_kernel(+ SMOKE_DIR,+ ("default",),+ unsloth_ref = "main",+ zoo_ref = "main",+ extra_args = (),+ per_run_timeout = 3600,+ )+ assert "EXPECTED_GPUS = 1" in "".join("".join(c["source"]) for c in other["cells"])+++def test_the_weights_go_on_one_card_while_both_stay_visible():+ """The two halves of the leg's configuration, and they pull opposite ways.++ Visibility is what makes unsloth's DEVICE_COUNT > 1 bindings live, so the+ leg must NOT be pinned. Placement is what makes training work, because a+ sharded model does not train: on unsloth-probe-multigpu-r1-18beab+ accelerate split Qwen3-0.6B across both T4s and step 0 died at+ unsloth/models/llama.py:972 with `index is on cuda:0, different from other+ tensors on cuda:1`.++ Dropping `--single-device` puts a red in front of every PR for an upstream+ fault; dropping `all_cards` silently turns the leg into a duplicate of+ Default. Both are asserted here because either alone reads as configured.+ """+ leg = legs.LEGS["multi_gpu"]+ assert leg.all_cards is True+ assert "--single-device" in leg.args+++def test_single_device_reaches_the_child_and_sets_a_device_map():+ import ast++ source = (SMOKE_DIR / "run_t4_smoke.py").read_text(encoding = "utf-8")+ tree = ast.parse(source)+ train = next(+ n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == "train_once"+ )+ body = ast.unparse(train)+ assert "single_device" in body+ assert "'device_map'] = {'': 0}" in body or '"device_map"] = {"": 0}' in body+ # The CYCLE loads the model, so a parent that parsed this and kept it would+ # shard anyway and the leg would die exactly as the probe did.+ #+ # The APPEND, not the string. `"--single-device" in unparse(main)` was the+ # first version of this and it survived deleting the forwarding outright,+ # because `ap.add_argument("--single-device", ...)` is in the same function+ # -- an assertion satisfied by its own surrounding text.+ main = next(n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == "main")+ assert "cmd.append('--single-device')" in ast.unparse(main), (+ "the flag is parsed but never forwarded, so the cycle child loads "+ "sharded and dies at step 0 exactly as the probe did"+ )+++def test_a_crash_mid_cycle_still_reports_what_was_measured():+ """The probe measured everything this leg asserts, logged it in full, and+ then LOST it: the cycle died in trainer.train(), wrote no report, and the+ leg reported `multi_gpu: null` while the driver log held every number.++ Reading the report alone said the measurement had not been taken, which is+ the difference between "we did not look" and "we looked and it was fine".+ """+ import ast++ source = (SMOKE_DIR / "run_t4_smoke.py").read_text(encoding = "utf-8")+ tree = ast.parse(source)+ facts = next(+ n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == "multi_gpu_facts"+ )+ # Stashed as soon as there is anything to stash, not on the way out: a+ # reading that lives only in a frame being unwound is a reading nobody has.+ published = ast.unparse(facts)+ assert "global _LAST_MULTI_GPU_FACTS" in published+ assert "_LAST_MULTI_GPU_FACTS = facts" in published++ main = next(n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == "main")+ child = ast.unparse(main)+ assert "except BaseException as exc" in child+ assert "'multi_gpu': _LAST_MULTI_GPU_FACTS" in child+ # A partial report must be impossible to mistake for a completed cycle, and+ # the crash must still propagate -- swallowing it would turn a dead cycle+ # into a green one.+ assert "'partial': True" in child+ assert "'cycle_error'" in child+ assert "\n raise" in child+++def test_the_stash_is_populated_by_the_real_function_before_it_can_fail():+ """Driven, not read. `multi_gpu_facts` walks the model, and a model that+ raises mid-walk must still leave the device count behind."""+ payload = _payload()+ payload._LAST_MULTI_GPU_FACTS = None++ class _Exploding:+ def named_parameters(self):+ raise RuntimeError("boom")++ def modules(self):+ return []++ facts = payload.multi_gpu_facts(_Exploding())+ assert payload._LAST_MULTI_GPU_FACTS is facts+ assert "device_count" in payload._LAST_MULTI_GPU_FACTSdiff --git a/tests/kaggle/test_naive_trl_compare.py b/tests/kaggle/test_naive_trl_compare.pynew file mode 100644index 00000000000..9c66c322d69--- /dev/null+++ b/tests/kaggle/test_naive_trl_compare.py@@ -0,0 +1,169 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""The plain-TRL control arm, and the ways a "comparison" proves nothing.++The rules under test are deliberately weak, and that is the point. Two library+stacks do not produce one fp16 trajectory -- `frontier` measured transformers+5.5.0 and 5.15.1 disagreeing at step 1 on identical weights, data and seed --+so a guard that asserts the arms AGREE would be red on ordinary drift. These+assert only what a comparison is entitled to: the control ran, it converged,+and it ran the same number of steps as the arm it is printed beside.+"""++from __future__ import annotations++import ast+import sys+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "t4_smoke"+sys.path.insert(0, str(PAYLOAD))++from naive_trl_compare import comparison_failures # noqa: E402+++def _trace(*losses):+ return {"metrics": [{"step": i + 1, "loss": v} for i, v in enumerate(losses)]}+++def test_a_control_that_never_ran_is_a_failure_not_a_silence():+ """The finding this file exists for. A missing arm must not read as a pass:+ "no comparison" and "the comparison agreed" are opposite outcomes and only+ one of them is evidence."""+ assert comparison_failures(None, [{"loss": 1.0}])+ assert comparison_failures({"error": "OOM"}, [{"loss": 1.0}])+ assert "did not run" in comparison_failures({"error": "OOM"}, None)[0]+++def test_a_control_that_loaded_and_trained_nothing_is_a_failure():+ broken = comparison_failures({"metrics": []}, [{"loss": 1.0}])+ assert len(broken) == 1 and "reported no steps" in broken[0]+++def test_a_converging_control_passes():+ assert comparison_failures(_trace(9.0, 5.0, 2.0), [{"loss": 1.0}] * 3) == []+++def test_a_flat_or_rising_control_fails():+ assert comparison_failures(_trace(2.0, 5.0, 9.0), [{"loss": 1.0}] * 3)+ assert comparison_failures(_trace(2.0, 2.0, 2.0), [{"loss": 1.0}] * 3)+++def test_a_non_finite_loss_fails_and_short_circuits():+ broken = comparison_failures(_trace(9.0, float("nan"), 2.0), [{"loss": 1.0}] * 3)+ assert len(broken) == 1 and "non-finite" in broken[0]+++def test_the_arms_must_have_run_the_same_number_of_steps():+ """A control that quietly ran fewer steps is printed beside a full unsloth+ trace as though the two were the same experiment."""+ broken = comparison_failures(_trace(9.0, 5.0, 2.0), [{"loss": 1.0}] * 10)+ assert broken and "different numbers of steps" in broken[0]+++def test_the_arms_are_never_asserted_equal():+ """Mutation-proof against the obvious "improvement". Two wildly different+ converging traces must PASS, because asserting agreement is what would make+ this check red on every ordinary version bump."""+ assert (+ comparison_failures(+ _trace(10.3222, 6.0, 1.0), [{"loss": 6.4367}, {"loss": 3.0}, {"loss": 0.5}]+ )+ == []+ )+++def test_the_control_module_never_imports_unsloth():+ """Asserted from the SOURCE, not from a convention. Anything that has+ imported unsloth has had transformers, trl and peft patched underneath it+ and is no longer a control; the comparison would be unsloth against itself+ with extra steps, and it would look exactly like a real result."""+ tree = ast.parse((PAYLOAD / "naive_trl_compare.py").read_text(encoding = "utf-8"))+ imported = set()+ for node in ast.walk(tree):+ if isinstance(node, ast.Import):+ imported.update(a.name.split(".")[0] for a in node.names)+ elif isinstance(node, ast.ImportFrom) and node.module:+ imported.add(node.module.split(".")[0])+ assert "unsloth" not in imported, sorted(imported)+ assert "unsloth_zoo" not in imported, sorted(imported)+++def test_the_payload_runs_the_control_in_a_separate_process():+ """A control imported into the parent would be patched by whatever the+ parent imported. It must be spawned, and it must be spawned AFTER the+ cycles: two 4bit models resident at once on a 14.56GB T4 is how a+ comparison becomes an OOM blamed on the thing being compared."""+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")+ assert "naive_trl_compare.py" in src+ assert "if args.compare_naive_trl:" in src+ cycles_at = src.index("runs.append(json.loads(report_file.read_text")+ spawn_at = src.index('"naive_trl_compare.py"')+ assert cycles_at < spawn_at, "the control arm must be spawned after the cycles"+++def test_the_control_arm_loads_the_repo_unsloth_resolved():+ """Not the name that was asked for, and the difference is an OOM.++ `load_in_4bit=True` sends unsloth through FLOAT_TO_INT_MAPPER to a+ pre-quantised `-unsloth-bnb-4bit` sibling. The plain path quantises the+ ORIGINAL on the fly and has to materialise the 16bit checkpoint first. On+ gemma-4-E2B-it that asked for 8.75 GiB on top of 7.25 GiB already resident+ and died (kernel unsloth-probe-latestcompile-r3-cb1125).++ Pointing both arms at the same weights is also the fairer comparison: the+ question is what the two training stacks do, not which repo each loader+ picks.+ """+ src = (ROOT / "tests" / "kaggle" / "t4_smoke" / "run_t4_smoke.py").read_text(encoding = "utf-8")+ assert 'control_model = runs[0].get("resolved_checkpoint") or args.model' in src+ assert '("--model", control_model),' in src+++def test_the_control_arm_uses_gradient_checkpointing():+ """Leaving it off was unfair rather than neutral.++ The unsloth arm runs with `gradient_checkpointing="unsloth"`, so a control+ without it is measured with the single largest memory lever disabled on one+ side only. On gemma-4-E2B-it that is the difference between a comparison and+ an OOM: the control asked for 8.75 GiB on top of 8.96 GiB already resident,+ on a 14.56 GiB card (kernel unsloth-probe-latestcompile-r4-e67ef2).+ """+ src = (PAYLOAD / "naive_trl_compare.py").read_text(encoding = "utf-8")+ assert "use_gradient_checkpointing = True" in src+ assert "gradient_checkpointing = True," in src+ # Non-reentrant, or a PEFT model's inputs carry no grad and the backward+ # fails with "element 0 of tensors does not require grad".+ assert 'gradient_checkpointing_kwargs = {"use_reentrant": False}' in src+++def test_a_load_time_oom_can_be_reported_rather_than_failed():+ """Measured: on gemma-4-E2B-it the plain arm asks for 8.75 GiB with 8.96 GiB+ already resident on a 14.56 GiB T4, at LOAD -- `metrics` is absent, so no+ step ever ran, and enabling gradient checkpointing changed the number not at+ all. That is a statement about the card and the checkpoint, not about either+ training stack."""+ oom = {"error": "OutOfMemoryError: CUDA out of memory. Tried to allocate 8.75 GiB"}+ assert comparison_failures(oom, [{"loss": 1.0}], allow_oom = True) == []+++def test_an_oom_is_still_a_failure_when_the_leg_did_not_opt_in():+ oom = {"error": "OutOfMemoryError: CUDA out of memory"}+ assert comparison_failures(oom, [{"loss": 1.0}])+++def test_an_oom_after_training_started_is_still_a_failure():+ """The narrowness is the point. An OOM DURING training is a finding about+ the run; only a failure to load is a fact about the card."""+ oom = {+ "error": "OutOfMemoryError: CUDA out of memory",+ "metrics": [{"step": 1, "loss": 3.0}],+ }+ assert comparison_failures(oom, [{"loss": 1.0}], allow_oom = True)+++def test_a_non_oom_crash_is_never_excused():+ crash = {"error": "ImportError: no module named trl"}+ assert comparison_failures(crash, [{"loss": 1.0}], allow_oom = True)diff --git a/tests/kaggle/test_prefetch_covers_the_wired_legs.py b/tests/kaggle/test_prefetch_covers_the_wired_legs.pynew file mode 100644index 00000000000..5c5eebd0538--- /dev/null+++ b/tests/kaggle/test_prefetch_covers_the_wired_legs.py@@ -0,0 +1,198 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""Every checkpoint a wired leg loads must be on the prefetch lane.++The prefetch runs in the driver's own interpreter from t=0, on no card and no+virtualenv, and it measured **~203 MB/s** on kernel+unsloth-probe-prefetch-verify-9568-7a0bdd -- 12.5 GB of gpt-oss in 61.7s. A repo+it does not name is downloaded by the leg instead, ON the card, while that card+is allocated and idle.++The failure is silent in the way this directory keeps being caught by: nothing+is red, the leg simply takes longer, and a schedule built on the assumption that+downloads are hidden is quietly wrong. It had already happened. `PREFETCH_REPOS`+listed Qwen2.5-0.5B-Instruct and gpt-oss, while `vision_fla_compile` --+the leg that SETS the makespan -- fetched its own 4.58 GB Qwen3.5-2B inline, and+`default` fetched Qwen3-0.6B inline.++Two details the rule has to respect, both learned the hard way:++* a leg with no ``--model`` takes its payload's argparse default, so reading the+ args alone reports the wrong checkpoint for every such leg;+* ``load_in_4bit=True`` sends unsloth through FLOAT_TO_INT_MAPPER to a+ ``-unsloth-bnb-4bit`` sibling, so warming the name in the args can warm a+ cache the leg never reads. ``LOAD_REDIRECTS`` records the ones that are known+ to differ.+"""++from __future__ import annotations++import re+import sys+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD_DIR = ROOT / "tests" / "kaggle" / "t4_smoke"+sys.path.insert(0, str(ROOT / ".github" / "scripts"))++from kaggle_t4_ci import legs # noqa: E402+++def _payload_default(entry: str) -> str | None:+ """The `--model` default the payload itself carries."""+ source = (PAYLOAD_DIR / entry).read_text(encoding = "utf-8")+ if 'ap.add_argument("--model", default = DEFAULT_MODEL)' not in source:+ return None+ match = re.search(r'^DEFAULT_MODEL = "([^"]+)"', source, re.MULTILINE)+ return match.group(1) if match else None+++def models_for(leg) -> set[str]:+ args = list(leg.args)+ if "--model" in args:+ named = args[args.index("--model") + 1]+ else:+ named = _payload_default(leg.entry)+ if not named:+ return set()+ # The declared name AND whatever it really resolves to on an sm_75 card.+ return {named, legs.LOAD_REDIRECTS.get(named, named)}+++def test_every_wired_leg_loads_a_prefetched_checkpoint():+ wired = [name for kernel in legs.KERNELS for name in kernel]+ missing = {}+ for name in wired:+ wanted = models_for(legs.LEGS[name])+ # The redirect target is what gets read, so satisfying either name is+ # enough only when they are the same repo.+ if wanted and not (wanted & set(legs.PREFETCH_REPOS)):+ missing[name] = sorted(wanted)+ assert not missing, (+ "these wired legs download their checkpoint on an allocated card "+ f"instead of on the free prefetch lane: {missing}. Add them to "+ "PREFETCH_REPOS."+ )+++def test_the_model_walk_reads_the_payload_default_and_not_only_the_args():+ """Three of the five wired legs carry no --model at all. A rule that read+ the args alone would report them as having no checkpoint and pass by+ finding nothing, which is the shape of a guard that guards nothing."""+ assert models_for(legs.LEGS["canary"]) == {"unsloth/Qwen2.5-0.5B-Instruct"}+ assert "unsloth/gpt-oss-20b-unsloth-bnb-4bit" in models_for(legs.LEGS["gptoss"]), (+ "the gpt-oss redirect is not being applied, so the prefetch would warm "+ "the 16-bit repo that no sm_75 run ever loads"+ )+++def test_the_critical_path_leg_is_fetched_before_the_one_with_slack():+ """Order is not decoration: the lane fetches in the order given.++ vision_fla_compile starts at t~21 and sets the makespan; gptoss is admitted+ only when a card empties, around t~500 on the measured schedule. The+ original order put gpt-oss first for margin when D was unknown; D is+ measured now, so the leg with the least slack goes first.+ """+ order = list(legs.PREFETCH_REPOS)+ assert order.index("unsloth/Qwen3.5-2B") < order.index(+ "unsloth/gpt-oss-20b-unsloth-bnb-4bit"+ ), "the leg that sets the makespan is queued behind the leg with 500s of slack"+++def test_nothing_is_prefetched_that_no_leg_reads():+ """The other direction, and it costs bandwidth rather than time: a repo+ nobody loads is a download the session pays for and never uses. Checked+ against every leg, not only the wired ones, so a leg parked in UNWIRED can+ keep its entry."""+ loaded = set()+ for leg in legs.LEGS.values():+ loaded |= models_for(leg)+ # Studio's own models are fetched by the Studio builder under its own+ # HF_HOME, so they are deliberately not in this list.+ stray = [repo for repo in legs.PREFETCH_REPOS if repo not in loaded]+ assert not stray, f"prefetched but never loaded by any leg: {stray}"+++def test_every_redirect_target_is_prefetched_under_its_EXACT_name():+ """The HF cache keys on the literal repo string.++ `models--unsloth--qwen3-0.6b-unsloth-bnb-4bit` and+ `models--unsloth--Qwen3-0.6B-unsloth-bnb-4bit` are different directories, so+ prefetching the pretty spelling of a repo the loader asks for in lower case+ warms a cache nobody reads and the session downloads it twice -- at full+ cost, with no error and nothing red. Two hardware reports give the exact+ strings; this asserts the lists agree with them character for character.+ """+ prefetch = set(legs.PREFETCH_REPOS)+ wired = {name for kernel in legs.KERNELS for name in kernel}+ for leg_name in wired:+ leg = legs.LEGS[leg_name]+ args = list(leg.args)+ named = (+ args[args.index("--model") + 1] if "--model" in args else _payload_default(leg.entry)+ )+ if named is None or named not in legs.LOAD_REDIRECTS:+ continue+ target = legs.LOAD_REDIRECTS[named]+ assert target in prefetch, (+ f"{leg_name} loads {target!r}, which is not prefetched under that "+ f"exact string; a case-different entry does not warm the same cache"+ )+++def test_the_lower_case_qwen3_redirect_is_not_tidied_away():+ """It looks like a typo and it is a measurement. Written down so the next+ reader corrects the capitals in the report, not in this file."""+ assert legs.LOAD_REDIRECTS["unsloth/Qwen3-0.6B"] == "unsloth/qwen3-0.6b-unsloth-bnb-4bit"+++def test_a_blob_is_counted_once_not_once_per_symlink(tmp_path, monkeypatch):+ """`snapshot_download` writes each file once under `blobs/` and links to it+ from `snapshots/`, so an `os.stat` walk follows the link and counts the same+ bytes twice.++ Measured on kernel unsloth-probe-prefetch-verify-9568-7a0bdd: gpt-oss came+ back as 25109731082 bytes for a ~12.5 GB checkpoint, and the reported+ 407.0 MB/s was really ~203. This is the number the second-wave ordering and+ the makespan argument rest on, so a 2x is not cosmetic.++ Drives the REAL generated cell body rather than a copy of the walk: the+ function under test only ever exists inside that f-string, and a+ reimplementation here would pass while the shipped one doubles.+ """+ import importlib.util+ import os++ spec = importlib.util.spec_from_file_location(+ "kaggle_prefetch_under_test", ROOT / ".github" / "scripts" / "kaggle_prefetch.py"+ )+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)++ folder = tmp_path / "hub" / "models--org--model"+ blobs = folder / "blobs"+ snapshot = folder / "snapshots" / "rev"+ blobs.mkdir(parents = True)+ snapshot.mkdir(parents = True)+ for index in range(3):+ blob = blobs / f"sha{index}"+ blob.write_bytes(b"x" * 1000)+ (snapshot / f"shard{index}.safetensors").symlink_to(blob)+ # A config is a real file in both places on a filesystem without symlinks,+ # which is the other way the same bytes get counted twice.+ (blobs / "cfg").write_bytes(b"y" * 10)+ os.link(blobs / "cfg", snapshot / "config.json")++ source = module.prefetch_cell(repos = [("org/model", None)], hf_home = str(tmp_path))+ match = re.search(r"def _repo_bytes\(repo\):.*?\n\ndef ", source, re.S)+ assert match, "the generated cell no longer defines _repo_bytes"+ namespace = {"os": os}+ exec(match.group(0)[: -len("\n\ndef ")], namespace) # noqa: S102++ monkeypatch.setenv("HF_HOME", str(tmp_path))+ assert namespace["_repo_bytes"]("org/model") == 3010, (+ "the walk counts a blob once per link to it, so every reported size and "+ "MB/s in the prefetch evidence is inflated"+ )diff --git a/tests/kaggle/test_studio_api_key.py b/tests/kaggle/test_studio_api_key.pynew file mode 100644index 00000000000..a428c442176--- /dev/null+++ b/tests/kaggle/test_studio_api_key.py@@ -0,0 +1,109 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""Minting a Studio API key, and the two ways that check proves nothing.++Creating a key proves the endpoint returns a string. Whether that string+authenticates anything is a separate question, and it is the one that matters:+a key Studio issues and then rejects fails in a user's integration rather than+here.++**Vacuity 1: passing on the session token.** If the bearer is not swapped, the+request succeeds because the session is still logged in and the key is never+exercised.++**Vacuity 2: a server that ignores the header.** Then any key "works", including+one that was never issued. The corrupted-key call is what rules that out, and+without it the positive check is satisfied by a server with no auth at all.+"""++from __future__ import annotations++import ast+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"+SRC = PAYLOAD.read_text(encoding = "utf-8")+++def _assert_api_key_body() -> str:+ tree = ast.parse(SRC)+ func = next(+ node+ for node in ast.walk(tree)+ if isinstance(node, ast.FunctionDef) and node.name == "assert_api_key"+ )+ return ast.get_source_segment(SRC, func) or ""+++def test_the_key_is_created_through_the_real_endpoint():+ body = _assert_api_key_body()+ assert '"/api/auth/api-keys"' in body+++def test_the_session_token_is_set_aside_while_the_key_is_driven():+ """Vacuity 1. Leaving the session token in place makes the call succeed for+ a reason that has nothing to do with the key."""+ body = _assert_api_key_body()+ assert "saved = self.studio.token" in body+ assert "self.studio.token = saved" in body, "the session token must be restored"++ # Structural, not a substring. `self.studio.token = raw_key[:-4] + "0000"`+ # CONTAINS the string "self.studio.token = raw_key", so a substring check+ # passes even when the swap line is deleted and only the corrupted-key+ # assignment remains -- which is exactly the mutation this guard exists to+ # catch, and it survived the first version of this test.+ tree = ast.parse(body)+ swaps = [+ node+ for node in ast.walk(tree)+ if isinstance(node, ast.Assign)+ and isinstance(node.value, ast.Name)+ and node.value.id == "raw_key"+ and any(isinstance(t, ast.Attribute) and t.attr == "token" for t in node.targets)+ ]+ assert swaps, (+ "the bearer is never set to the raw key itself, so the positive check "+ "runs on whatever token was already there"+ )+++def test_a_corrupted_key_must_be_rejected():+ """Vacuity 2. Without this, a server that ignores the Authorization header+ entirely passes the positive check."""+ body = _assert_api_key_body()+ assert "bad_key_status" in body+ assert "a corrupted API key was accepted" in body+++def test_the_raw_key_is_registered_as_a_secret_before_it_is_used():+ """It reaches request headers and, on failure, error strings that travel+ home in the evidence bundle."""+ body = _assert_api_key_body()+ add_at = body.index("self.secrets.add(raw_key)")+ use_at = body.index("self.studio.token = raw_key")+ assert add_at < use_at, "the key is used before it is registered for scrubbing"+++def test_the_session_token_is_restored_even_when_the_call_raises():+ """A finally, not a trailing statement: an exception mid-check would+ otherwise leave the session authenticated as a corrupted API key and every+ later assertion would fail for the wrong reason."""+ tree = ast.parse(_assert_api_key_body())+ tries = [n for n in ast.walk(tree) if isinstance(n, ast.Try) and n.finalbody]+ restores = [+ t+ for t in tries+ if "self.studio.token = saved" in "".join(ast.dump(s) for s in t.finalbody)+ or any("saved" in ast.dump(s) for s in t.finalbody)+ ]+ assert restores, "the token restore is not in a finally"+++def test_it_runs_before_the_long_gpu_work():+ """It needs only a logged-in session. After a 20-minute training run, a+ training failure would hide whether API keys work at all."""+ api_at = SRC.index("self.assert_api_key()")+ train_at = SRC.index("trained = self.assert_training()")+ assert api_at < train_atdiff --git a/tests/kaggle/test_studio_cli_run.py b/tests/kaggle/test_studio_cli_run.pynew file mode 100644index 00000000000..a95c6ca6c1b--- /dev/null+++ b/tests/kaggle/test_studio_cli_run.py@@ -0,0 +1,221 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""`unsloth run`: the headless model server, and what a banner does not prove.++`unsloth run` is a different launch from `unsloth studio`, and nothing in CI+covered it. It starts the backend, waits for health, mints an API key+IN-PROCESS, and only then loads the model over HTTP. Any of those four steps+can fail while the command still prints a banner, so the rules here are about+what came back rather than what was printed.++The two that carry the most weight, because each closes a hole the other+leaves:++**GPU residency across the launch.** A GGUF server that fell back to the CPU+serves text perfectly well, so "it answered" is not evidence it reached the+card. The measurement is PER-PROCESS rather than a device total: under+--studio-concurrent a training leg shares the card, and on kernel+unsloth-probe-full-concurrent-417238 the device delta read -182.0 MiB while+the same report showed this launch holding 2628 MiB. A shared counter cannot+attribute. `cli_run_gpu_failure` is a pure function precisely so the rules+below can DRIVE it with those numbers rather than describe it.++**A corrupted key must be REFUSED.** Without it, a server that ignores the+Authorization header entirely satisfies "the minted key authenticated".+"""++from __future__ import annotations++import ast+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"+SRC = PAYLOAD.read_text(encoding = "utf-8")+++def _func(name: str) -> ast.FunctionDef:+ for cls in ast.walk(ast.parse(SRC)):+ if not isinstance(cls, ast.ClassDef):+ continue+ for node in cls.body:+ if isinstance(node, ast.FunctionDef) and node.name == name:+ return node+ raise AssertionError(f"no method named {name!r}")+++def _body(name: str = "assert_cli_run") -> str:+ return ast.get_source_segment(SRC, _func(name)) or ""+++def test_the_assertion_exists_and_is_driven_from_the_run():+ assert _body()+ assert "self.assert_cli_run()" in _body("execute")+++def test_it_runs_after_the_ui_phase_has_stopped_the_server():+ """Not a preference. `unsloth run` starts a SECOND backend against the same+ studio home, and two backends sharing one home's state is a configuration+ nobody ships. It also makes the VRAM delta meaningless: a card still+ holding the first server's model cannot show this launch's growth."""+ body = _body("execute")+ ui_at = body.index("self.assert_chat_ui()")+ cli_at = body.index("self.assert_cli_run()")+ assert ui_at < cli_at, "the CLI launch must come after the UI phase stops the server"+++def test_it_uses_its_own_port():+ """The first server's port may still be in TIME_WAIT, and a bind failure+ there would read as a broken CLI."""+ assert "port = self.args.port + 1" in _body()+++def test_no_public_url_is_ever_opened_from_ci():+ """--secure implies a Cloudflare quick tunnel, which publishes this server+ to the internet from a CI kernel. --no-cloudflare is explicit rather than+ relying on the default, because a default is a thing that changes."""+ body = "".join(_body().split()) # formatter-proof; see the cloudflare guards+ assert '"--no-cloudflare",' in body+ assert '"--secure"' not in body+++def test_the_key_comes_from_the_marker_the_cli_itself_prints():+ body = "".join(_body().split())+ assert '"--start-api-key-marker",' in body+ assert '"UNSLOTH_START_API_KEY:"intext' in body # whitespace-stripped+++def test_the_key_is_registered_as_a_secret_before_anything_reads_the_log():+ """The log is packed into the evidence bundle. `redacted()` is what keeps+ the key out of it, and it can only redact a secret it has been told about,+ so the registration has to happen at the moment the key is parsed."""+ src = _body()+ parse_at = src.index('text.split("UNSLOTH_START_API_KEY:"')+ add_at = src.index("self.secrets.add(api_key)")+ assert parse_at < add_at, "the key must be registered where it is parsed"+ # And the log must actually be in the bundle, or nothing redacts it.+ assert '"unsloth_run.log",' in _body("emit_evidence")+++def test_gpu_use_is_measured_and_an_unmeasurable_reading_is_a_failure():+ """ "nvidia-smi did not answer" and "the model was on the GPU" are opposite+ outcomes; treating the first as a pass is the exact shape this directory+ has been caught by before."""+ body = _body()+ assert "baseline = nvidia_used_mib()" in body+ assert "settled = nvidia_used_mib()" in body+ # The verdict itself moved into `cli_run_gpu_failure` when the device delta+ # stopped being a valid ruler under --studio-concurrent, so it is DRIVEN+ # rather than grepped -- see the rules at the end of this file. What stays+ # here is that the assertion consults it and reports what it says.+ assert "cli_run_gpu_failure(" in body+ assert "failures.append(failure)" in body+ verdict = _verdict()+ assert verdict(None, None, None, None)[0], "an unmeasurable reading passed"+++def test_a_corrupted_key_must_be_refused():+ """Without this, a server ignoring the Authorization header entirely+ satisfies the claim that the minted key authenticated."""+ func = _func("assert_cli_run")+ guarded = [+ n+ for n in ast.walk(func)+ if isinstance(n, ast.If)+ and "bad_key_status" not in ast.unparse(n.test)+ and "code < 400" in ast.unparse(n.test)+ ]+ assert guarded, "nothing refuses a corrupted key"+++def test_the_child_is_always_torn_down():+ """A `unsloth run` left alive holds a card and a port for the rest of the+ session, and the kernel's next phase reads that as its own failure."""+ func = _func("assert_cli_run")+ tries = [n for n in ast.walk(func) if isinstance(n, ast.Try) and n.finalbody]+ assert tries, "the teardown must be in a finally, or a raised assertion leaks the server"+ finals = "\n".join(ast.unparse(n) for t in tries for n in t.finalbody)+ assert "proc.terminate()" in finals+ assert "proc.kill()" in finals, "terminate alone leaves a hung server running"+++def test_the_vram_sample_comes_AFTER_a_served_completion():+ """`unsloth run` prints its API key while it is still starting.++ Sampling there read 0.0 MiB of growth on kernel+ unsloth-probe-studio-full2-815a0c, on a launch whose own log says+ `Starting llama-server: ... -ngl -1 --fit off` -- Studio asking for every+ layer on the card. A completion that came back is the cheap proof the+ weights are resident, so the ruler has to go after it or the check measures+ a race.+ """+ func = _func("assert_cli_run")+ src = ast.get_source_segment(SRC, func) or ""+ sample_at = src.index('detail["vram_after_mib"]')+ completion_at = src.index('detail["completion_status"]')+ assert completion_at < sample_at, (+ "VRAM is sampled before a completion has been served, so a slow load "+ "reads as a CPU fallback"+ )+++def _verdict():+ """The real function, loaded by path rather than reimplemented."""+ import importlib.util++ spec = importlib.util.spec_from_file_location("_studio_payload_cli", PAYLOAD)+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return module.cli_run_gpu_failure+++def test_a_co_tenant_freeing_memory_does_not_read_as_a_CPU_fallback():+ """The exact numbers from unsloth-probe-full-concurrent-417238.++ Device VRAM 2816 -> 2634, a delta of -182.0, while `nvidia-smi` shows this+ launch's own pid holding 2628 MiB. Under the old device-delta rule that was+ a failure saying `unsloth run` served from the CPU; the model was on the+ card the whole time and a training leg on the same card freed memory inside+ the window. This is the regression guard for that reading.+ """+ failure, detail = _verdict()({}, {6841: 2628}, 2816.0, 2634.0)+ assert failure is None, failure+ assert detail["process_vram_mib"] == 2628+ # The device delta is still RECORDED -- it is evidence, it is just not the+ # verdict -- and it is still the number that misled.+ assert detail["vram_delta_mib"] == -182.0+++def test_a_co_tenant_ALREADY_on_the_card_cannot_satisfy_the_claim():+ """Counting every process would pass on a card a training leg is using and+ a server that never left the CPU. Only pids that APPEARED count."""+ failure, detail = _verdict()({99: 12000}, {99: 12000}, 100.0, 100.0)+ assert failure and "served from the CPU" in failure+ assert detail["process_vram_mib"] == 0+++def test_a_real_cpu_fallback_still_fails():+ """The case the assertion exists for: the launch answered, and no process+ of its own ever appeared on the GPU."""+ failure, _ = _verdict()({99: 12000}, {99: 12000, 4242: 3}, 100.0, 101.0)+ assert failure and "served from the CPU" in failure+++def test_the_device_delta_is_the_fallback_only_when_processes_are_unreadable():+ """An nvidia-smi that answers a total but cannot enumerate apps still gets a+ verdict rather than a silent pass."""+ verdict = _verdict()+ assert verdict(None, None, 100.0, 4000.0)[0] is None+ failure, _ = verdict(None, None, 100.0, 110.0)+ assert failure and "could not enumerate processes" in failure+ assert verdict(None, None, None, None)[0] == (+ "nvidia-smi did not answer, so GPU use is unmeasured"+ )+++def test_the_before_sample_is_taken_before_the_launch():+ """An `apps_before` read after the server started would contain the server,+ so nothing would ever have `appeared` and every run would fail."""+ body = _body()+ assert body.index("apps_before = nvidia_compute_apps()") < body.index("subprocess.Popen")diff --git a/tests/kaggle/test_studio_cloudflare.py b/tests/kaggle/test_studio_cloudflare.pynew file mode 100644index 00000000000..4aaeb40d3fb--- /dev/null+++ b/tests/kaggle/test_studio_cloudflare.py@@ -0,0 +1,163 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""The quick tunnel: the one assertion here that reaches the public internet.++Because it does, what it claims is narrow and what it refuses to claim is+written down.++Three rules, and the third is the one that makes opening a public URL from a CI+machine defensible at all: the tunnel must REFUSE an unauthenticated request.+Without it this check would prove that anyone holding the URL can drive+inference on the box, and call that a pass.++Two details are load-bearing rather than stylistic:++* `--host 0.0.0.0`. Studio raises a quick tunnel for WILDCARD binds only; on+ 127.0.0.1 there is nothing to publish, no URL is printed, and a working+ feature reads as a broken one.+* the negative lookahead on `api.trycloudflare.com`. That host appears in+ cloudflared's own FAILURE lines ("failed to request quick Tunnel: Post+ https://api.trycloudflare.com/tunnel"), so a naive match extracts a URL from+ the message that says there is no URL. Studio's own matcher carries the same+ lookahead, and this one is checked against both strings.+"""++from __future__ import annotations++import ast+import re+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"+SRC = PAYLOAD.read_text(encoding = "utf-8")+++def _func(name: str) -> ast.FunctionDef:+ for cls in ast.walk(ast.parse(SRC)):+ if not isinstance(cls, ast.ClassDef):+ continue+ for node in cls.body:+ if isinstance(node, ast.FunctionDef) and node.name == name:+ return node+ raise AssertionError(f"no method named {name!r}")+++def _body(name: str = "assert_cloudflare") -> str:+ return ast.get_source_segment(SRC, _func(name)) or ""+++def test_the_assertion_exists_and_is_driven_from_the_run():+ assert _body()+ assert "self.assert_cloudflare()" in _body("execute")+++def _flat(text: str) -> str:+ """Whitespace-stripped, because the repo's formatter reflows an argument+ list to one entry per line and a guard matching the unformatted spelling+ goes red on a reformat rather than on a regression. That has happened three+ times in this payload now."""+ return "".join(text.split())+++def test_it_binds_a_wildcard_or_there_is_nothing_to_publish():+ body = _flat(_body())+ assert '"--host","0.0.0.0",' in body+ assert '"--cloudflare",' in body+++def test_the_url_matcher_rejects_cloudflareds_own_failure_host():+ """Executed, not eyeballed. The failure line contains a trycloudflare URL+ and means the opposite of a tunnel."""+ body = _body()+ literal = re.search(r"re\.search\((r\"[^\"]+\")", body)+ assert literal, "no URL matcher found"+ pattern = re.compile(ast.literal_eval(literal.group(1)))+ assert pattern.search("Your quick Tunnel: https://brave-cat-runs.trycloudflare.com")+ assert not pattern.search(+ 'failed to request quick Tunnel: Post "https://api.trycloudflare.com/tunnel"'+ ), "the failure host was accepted as a working tunnel URL"+++def test_the_tunnel_must_actually_serve():+ """ "a URL was printed" and "a URL that works" are different claims. A+ tunnel whose edge never registers answers 530."""+ func = _func("assert_cloudflare")+ assert any(+ isinstance(n, ast.If) and ast.unparse(n.test) == "code != 200" for n in ast.walk(func)+ ), "nothing checks that the public URL answers"+++def test_an_unauthenticated_request_through_the_tunnel_must_be_refused():+ """The rule that makes opening the URL defensible. Asserted structurally so+ a message rewrite cannot satisfy it."""+ func = _func("assert_cloudflare")+ unauth = [+ n+ for n in ast.walk(func)+ if isinstance(n, ast.Call)+ and any(k.arg == "auth" and k.value.value is False for k in n.keywords)+ ]+ assert len(unauth) >= 2, (+ "both the health probe and the inference probe must go through the "+ "tunnel WITHOUT credentials"+ )+ assert any(+ isinstance(n, ast.If) and "code < 400" in ast.unparse(n.test) for n in ast.walk(func)+ ), "an accepted unauthenticated request must be a failure"+++def test_the_no_tunnel_excuse_is_narrow():+ """Reported-not-failed applies ONLY when nothing was published, because+ then there is nothing to have gone wrong with. Once a URL exists, every+ rule above is live."""+ func = _func("assert_cloudflare")+ excuses = [+ n for n in ast.walk(func) if isinstance(n, ast.If) and ast.unparse(n.test) == "url is None"+ ]+ assert excuses, "the excuse branch is gone or is no longer gated on the URL"+ for node in excuses:+ # node.body only. Walking the whole If would sweep in the `else`+ # branch, which is where every real failure lives, and the guard would+ # then fail on correct code -- it did, the first time this was written.+ appended = [+ n+ for stmt in node.body+ for n in ast.walk(stmt)+ if isinstance(n, ast.Call)+ and isinstance(n.func, ast.Attribute)+ and n.func.attr == "append"+ and isinstance(n.func.value, ast.Name)+ and n.func.value.id == "failures"+ ]+ assert not appended, "the no-tunnel branch reports; it does not fail"+ body = _body()+ assert "no_tunnel_reason" in body, "a reason must be carried from the log"+++def test_the_public_url_is_registered_as_a_secret():+ """It is a live public route to this machine and the artifact is read by+ people who are not running it."""+ body = _body()+ assert "self.secrets.add(url)" in body+ assert '"unsloth_cloudflare.log",' in _body(+ "emit_evidence"+ ), "a log nobody packs is a log nobody redacts"+++def test_the_child_is_always_torn_down():+ """A tunnel left up outlives the assertion and keeps the machine public."""+ func = _func("assert_cloudflare")+ finals = "\n".join(+ ast.unparse(n) for t in ast.walk(func) if isinstance(t, ast.Try) for n in t.finalbody+ )+ assert "proc.terminate()" in finals+ assert "proc.kill()" in finals+++def test_it_can_be_switched_off():+ """The only assertion here that reaches the public internet should be+ refusable without editing the payload."""+ assert '"--no-cloudflare-check"' in SRC+ assert "self.args.cloudflare_check" in _body("execute")diff --git a/tests/kaggle/test_studio_code_execution.py b/tests/kaggle/test_studio_code_execution.pynew file mode 100644index 00000000000..4df1b86bea7--- /dev/null+++ b/tests/kaggle/test_studio_code_execution.py@@ -0,0 +1,165 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""Studio's local `python` tool must be EXECUTED, not merely offered.++The vacuity this file exists against is a specific one. `assert_tool_calling`+already proves the model can emit a tool call, and a weather tool is never run+by Studio at all -- the caller runs it. The local `python` tool is different:+Studio executes it in a per-session sandbox, and the failure worth catching is+a loop that hands the model the schema, never runs the call, and lets the model+narrate a plausible result. From the reply text that is indistinguishable from+success, because a language model will happily report the output of code that+never ran.++So the pass condition must come off the FILESYSTEM. A per-run token is written+by the executed code into `<studio home>/sandbox`, and only real execution puts+those bytes on this disk.++One configuration detail is load-bearing rather than incidental:+`routes/inference.py` rejects a local python/terminal tool with a 400 under+`permission_mode` `ask`, and under `auto` or the omitted default, because there+is no confirmation channel on this path. A payload that left it unset would+fail on configuration and read as a broken tool.+"""++from __future__ import annotations++import ast+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"+SRC = PAYLOAD.read_text(encoding = "utf-8")+++def _func(name: str) -> ast.FunctionDef:+ """The METHOD of that name, not a module-level function.++ `run` is both: a module-level `subprocess` helper and the harness's own+ sequencer. Walking the whole tree finds the helper first, and every+ assertion about the sequence then reads a body that never mentions it.+ """+ tree = ast.parse(SRC)+ for cls in ast.walk(tree):+ if not isinstance(cls, ast.ClassDef):+ continue+ for node in cls.body:+ if isinstance(node, ast.FunctionDef) and node.name == name:+ return node+ raise AssertionError(f"no method named {name!r}")+++def _body(name: str = "assert_code_execution") -> str:+ return ast.get_source_segment(SRC, _func(name)) or ""+++def test_the_assertion_exists_and_is_driven_from_the_run():+ body = _body()+ assert body+ run = _body("execute")+ assert "self.assert_code_execution()" in run, "an assertion nothing calls is not coverage"+++def test_the_python_tool_is_the_one_requested():+ """`enabled_tools` omitted means ALL local tools, which would let the model+ satisfy this with `terminal` and say nothing about `python`."""+ body = _body()+ assert "enable_tools = True" in body+ assert 'enabled_tools = ["python"]' in body+++def test_the_permission_mode_is_one_the_local_python_tool_survives():+ """Not a style choice. `ask` is rejected outright for a local python tool,+ and so are `auto` and the omitted default, with a 400 -- there is no+ confirmation channel on this path. Either of those would fail the run on+ configuration while looking like a broken tool."""+ body = _body()+ assert 'permission_mode = "off"' in body+ assert 'permission_mode = "ask"' not in body+ assert 'permission_mode = "auto"' not in body+++def test_the_token_is_minted_per_run_and_not_hardcoded():+ """A fixed token would be satisfied by a file an EARLIER run left in the+ sandbox, which is a green tick for code that did not run this time."""+ body = _body()+ assert "secrets_module.token_hex" in body+ call = next(+ node+ for node in ast.walk(_func("assert_code_execution"))+ if isinstance(node, ast.Call)+ and isinstance(node.func, ast.Attribute)+ and node.func.attr == "token_hex"+ )+ assert call is not None+++def test_the_verdict_is_read_off_the_filesystem_and_not_off_the_reply():+ """The rule this whole file is about. The failure must be raised by the+ absence of a written file; a check on the reply text would pass on a model+ that narrated an execution that never happened.++ Asserted structurally rather than by message matching, because a message+ is satisfied by its own surrounding text.+ """+ func = _func("assert_code_execution")++ # Every `failures.append` that decides the verdict, and what it is guarded by.+ guarded_by_written = False+ for node in ast.walk(func):+ if not isinstance(node, ast.If):+ continue+ test = ast.unparse(node.test)+ appends = [+ n+ for n in ast.walk(node)+ if isinstance(n, ast.Call)+ and isinstance(n.func, ast.Attribute)+ and n.func.attr == "append"+ ]+ if not appends:+ continue+ if "written" in test:+ guarded_by_written = True+ assert isinstance(node.test, ast.UnaryOp) and isinstance(+ node.test.op, ast.Not+ ), "the failure must fire when NO file carries the token"+ assert guarded_by_written, (+ "nothing in this assertion depends on a file being written, so it "+ "would pass on a reply that merely claimed the code ran"+ )++ # And the reply is recorded but never judged.+ for node in ast.walk(func):+ if isinstance(node, ast.If) and "reply" in ast.unparse(node.test):+ raise AssertionError(+ "the verdict branches on the model's prose, which is exactly "+ "what an unexecuted tool call can fake"+ )+++def test_the_search_reads_content_rather_than_only_a_filename():+ """A small local model rewording the path is not a Studio defect. What+ cannot be faked is the token's BYTES landing on disk, so the content is+ what the search is entitled to rely on."""+ body = _body()+ assert "read_text" in body+ assert "token in body" in body+++def test_it_looks_under_the_studio_home_sandbox_and_not_a_stray_root():+ """`sandbox_root()` puts the per-session working directories under the+ studio home precisely so `UNSLOTH_STUDIO_HOME` keeps them together. Reading+ a fixed `~/studio_sandbox` would search a directory this run never wrote+ to, and report a correct execution as a failure."""+ body = _body()+ assert 'self.studio_home / "sandbox"' in body+++def test_the_cpu_fallback_records_the_assertion_rather_than_omitting_it():+ """A skipped assertion that leaves no record is indistinguishable from one+ that never existed, and the summary would be short by one line rather than+ red."""+ run = _body("execute")+ assert '"code_execution",' in run, "the non-GPU branch must record code_execution explicitly"diff --git a/tests/kaggle/test_studio_compaction.py b/tests/kaggle/test_studio_compaction.pynew file mode 100644index 00000000000..4429f55f22f--- /dev/null+++ b/tests/kaggle/test_studio_compaction.py@@ -0,0 +1,244 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""Context compaction, and why one half of the check is worthless alone.++Studio reports the fit on the completion itself, as `context_truncated` with+`dropped_messages`, so a conversation past the window can be shown to have been+SHORTENED rather than refused, without a browser.++The rule is a pair, deliberately:++* an over-length conversation returns 200 with `dropped_messages > 0`;+* a two-message conversation drops nothing.++The first alone passes on a server that reports truncation unconditionally,+which is indistinguishable from working. The second alone passes on a server+that never compacts and returns a context-length error instead. Only together+do they say the field tracks length.++It also has to run where the window is KNOWN. `assert_server_flags` reloads the+model pinned to `--studio-ctx`; a compaction check before that is aimed at a+context length nobody set.+"""++from __future__ import annotations++import ast+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"+SRC = PAYLOAD.read_text(encoding = "utf-8")+++def _func(name: str) -> ast.FunctionDef:+ for cls in ast.walk(ast.parse(SRC)):+ if not isinstance(cls, ast.ClassDef):+ continue+ for node in cls.body:+ if isinstance(node, ast.FunctionDef) and node.name == name:+ return node+ raise AssertionError(f"no method named {name!r}")+++def _body(name: str = "assert_compaction") -> str:+ return ast.get_source_segment(SRC, _func(name)) or ""+++def test_the_assertion_exists_and_is_driven_from_the_run():+ assert _body()+ assert "self.assert_compaction()" in _body("execute")+++def test_it_runs_after_the_window_is_pinned():+ body = _body("execute")+ flags_at = body.index("self.assert_server_flags()")+ ours_at = body.index("self.assert_compaction()")+ assert flags_at < ours_at, (+ "the reload in assert_server_flags is what pins the context length; "+ "before it, this check overflows a window nobody set"+ )+++def test_the_over_length_case_must_report_dropped_messages():+ func = _func("assert_compaction")+ tests = [ast.unparse(n.test) for n in ast.walk(func) if isinstance(n, ast.If)]+ assert any(+ "long_dropped" in t for t in tests+ ), "nothing depends on the long conversation having been shortened"+++def test_the_short_control_is_present_and_is_the_opposite_claim():+ """Without it, a server that always claims truncation passes."""+ func = _func("assert_compaction")+ short = [+ n+ for n in ast.walk(func)+ if isinstance(n, ast.If) and "short_dropped" in ast.unparse(n.test)+ ]+ assert short, "there is no negative control"+ for node in short:+ # It must fail when the short chat DID drop, which is the opposite+ # polarity to the long one.+ assert not (+ isinstance(node.test, ast.UnaryOp) and isinstance(node.test.op, ast.Not)+ ), "the control must fire when a SHORT conversation reports a drop"+++def test_a_refusal_is_a_failure_rather_than_a_pass():+ """A context-length error is not compaction. The distinction is the whole+ feature: one shortens the prompt and answers, the other gives up.++ Asserted structurally. The first version of this test matched the failure+ MESSAGE, with an `or` that was true of any body at all -- a guard satisfied+ by its own surrounding text, which is the exact shape this directory has+ been caught by five times.+ """+ func = _func("assert_compaction")+ refusals = [+ n+ for n in ast.walk(func)+ if isinstance(n, ast.If)+ and ast.unparse(n.test) == "code != 200"+ and any(+ isinstance(c, ast.Call)+ and isinstance(c.func, ast.Attribute)+ and c.func.attr == "append"+ for c in ast.walk(n)+ )+ ]+ assert len(refusals) >= 2, (+ "both the long conversation and the short control must fail on a "+ "non-200, or a server that refuses everything reads as a pass"+ )+++def test_the_overflow_is_built_past_the_budget_and_not_merely_to_it():+ """The budget subtracts the reply reserve and the template's own framing,+ so a prompt equal to the context length is not reliably over it."""+ body = _body()+ assert "for i in range(40):" in body+ assert "self.args.studio_ctx" in body+++def test_the_cpu_fallback_records_the_assertion_rather_than_omitting_it():+ assert '"compaction",' in _body("execute")+++def test_every_status_read_comes_from_a_real_call():+ """Found by mutation, not by reasoning: assigning `code = 200` after the+ request survived every other rule here, because they all read the shape of+ the branches and none of them asked where the value came from.++ A hardcoded status is unlikely as drift and trivial as a "fix" for a red,+ which is the same thing.+ """+ func = _func("assert_compaction")+ for node in ast.walk(func):+ if not isinstance(node, ast.Assign):+ continue+ targets = {t.id for t in node.targets if isinstance(t, ast.Name)}+ if not ({"code"} & targets or {"code", "body"} <= _tuple_names(node)):+ continue+ assert isinstance(node.value, ast.Call), (+ f"`code` is assigned from {ast.unparse(node.value)!r} rather than "+ f"from a request, so the status this checks is not the server's"+ )+++def _tuple_names(node: ast.Assign) -> set:+ out = set()+ for target in node.targets:+ if isinstance(target, ast.Tuple):+ out.update(e.id for e in target.elts if isinstance(e, ast.Name))+ return out+++def test_compaction_is_REQUESTED_rather_than_expected_by_default():+ """Measured on kernel unsloth-probe-studio-full2-815a0c, where this+ assertion failed a documented default.++ `context_overflow` defaults to "error": an over-length conversation comes+ back 400 with code=context_length_exceeded, so a client's own trim loop can+ see it. Compaction is a policy you ASK for. "truncate_oldest" is the one+ that applies to a plain chat; "truncate_middle" is limited to client-tool+ and response_format passthrough+ (studio/backend/models/inference.py:context_overflow).+ """+ body = _body()+ assert 'context_overflow = "truncate_oldest"' in body+ assert (+ 'context_overflow = "truncate_middle"' not in body+ ), "truncate_middle does not apply to a plain chat, so it would report a policy that never ran"+++def test_the_default_policy_control_is_present_and_expects_a_refusal():+ """The half that stops the request field from being decorative.++ Without it, the check above passes on a server that compacts everything+ regardless of what was asked for, and naming the policy proves nothing.+ """+ func = _func("assert_compaction")+ refusals = [+ node+ for node in ast.walk(func)+ if isinstance(node, ast.Compare) and "400" in ast.unparse(node)+ ]+ assert refusals, "nothing asserts the default policy still refuses"+ assert "long_status_default_policy" in _body()+++def test_the_refusal_is_checked_by_CODE_and_not_by_the_status_alone():+ """400 is not the claim. `openai_error_body` always emits the `code` key and+ leaves it None for an unclassified failure, so a validation error and a+ code-less generation error are both 400 -- and a status-only check accepts+ either as proof of overflow semantics the failure text promises.++ `code=context_length_exceeded` is what a client's trim loop detects+ (routes/inference.py, `_openai_stream_error_chunk`), so it is the assertion.+ """+ func = _func("assert_compaction")+ body = ast.unparse(func)+ # `ast.unparse` normalises quoting, so match the value not the literal.+ assert (+ "context_length_exceeded" in body+ ), "the default-policy control never reads the error code, so any 400 passes"+ compares = [+ ast.unparse(node)+ for node in ast.walk(func)+ if isinstance(node, ast.Compare) and "context_length_exceeded" in ast.unparse(node)+ ]+ assert compares, "context_length_exceeded appears but is never compared against"+ # And it has to gate the SAME branch as the status, or a run can refuse with+ # the wrong code and still pass.+ guards = [+ ast.unparse(node.test)+ for node in ast.walk(func)+ if isinstance(node, ast.If) and "context_length_exceeded" in ast.unparse(node.test)+ ]+ assert any("400" in guard for guard in guards), (+ "the code and the status are checked in separate branches, so one can "+ "pass while the other is wrong"+ )+++def test_the_over_length_conversation_is_sent_twice_under_two_policies():+ """One request cannot compare two policies. Both calls have to exist, and+ they have to send the SAME messages, or the pair compares a policy against+ a different conversation."""+ func = _func("assert_compaction")+ calls = [+ node+ for node in ast.walk(func)+ if isinstance(node, ast.Call)+ and isinstance(node.func, ast.Attribute)+ and node.func.attr == "chat"+ and "long_messages" in ast.unparse(node)+ ]+ assert len(calls) == 2, f"expected two long-conversation calls, found {len(calls)}"+ overflow = [call for call in calls if any(kw.arg == "context_overflow" for kw in call.keywords)]+ assert len(overflow) == 1, (+ "exactly one of the two must name the policy: both naming it removes "+ "the control, neither naming it removes the feature under test"+ )diff --git a/tests/kaggle/test_studio_gpu_harness.py b/tests/kaggle/test_studio_gpu_harness.pyindex 72bcf14fa21..2b81a86afa6 100644--- a/tests/kaggle/test_studio_gpu_harness.py+++ b/tests/kaggle/test_studio_gpu_harness.py@@ -19,7 +19,6 @@ Whichever ran first would win.from __future__ import annotations-import argparseimport base64import importlib.utilimport io@@ -994,11 +993,28 @@ def test_the_gate_and_launcher_are_the_shared_ones():# ------------------------------------------------------------------ report+_PASSING = [{"label": "studio-gpu", "passed": True, "assertions": []}]+_FAILING = [{"label": "studio-gpu", "passed": False, "failures": ["x"], "assertions": []}]+# A training leg from the merged kernel. Not this reporter's payload.+_LEG = [{"label": "control", "passed": False, "steps": []}]++@pytest.mark.parametrize(- ("verdict", "expected_exit"),- [("pass", 0), ("partial", 0), ("infra", 0), ("fail", 1)],+ ("verdict", "reports", "expected_exit"),+ [+ ("pass", _PASSING, 0),+ ("partial", [], 0),+ ("infra", [], 0),+ ("fail", _FAILING, 1),+ # The kernel verdict is the WHOLE kernel's, and since --with-studio+ # that kernel also carries four training legs. A leg failing must not+ # print "Studio GPU smoke: FAIL" over a passing Studio payload and send+ # someone to read the wrong half: the T4 reporter is what turns that+ # red, in the same job, from the same evidence directory.+ ("fail", _PASSING + _LEG, 0),+ ],)-def test_only_a_real_assertion_failure_turns_the_job_red(tmp_path, verdict, expected_exit):+def test_only_a_real_assertion_failure_turns_the_job_red(tmp_path, verdict, reports, expected_exit):evidence = tmp_path / "evidence"evidence.mkdir()(evidence / "launch_result.json").write_text(@@ -1008,7 +1024,7 @@ def test_only_a_real_assertion_failure_turns_the_job_red(tmp_path, verdict, expe"reason": "test","slug": "u/s","kernel_state": "COMPLETE",- "reports": [],+ "reports": reports,}))@@ -1225,11 +1241,22 @@ def _load_payload():def _session(module, tmp_path, **overrides):- args = argparse.Namespace(- repo_root = str(tmp_path / "repo"),- studio_home = str(tmp_path / "home"),- **overrides,+ # Built from the REAL parser rather than a hand-listed Namespace. A+ # hand-listed one carries exactly the attributes someone remembered, so+ # every new flag breaks these tests with an AttributeError that says+ # nothing about the flag -- which is what --studio-password did.+ args = module.parse_args(+ [+ "--outdir",+ str(tmp_path / "out"),+ "--repo-root",+ str(tmp_path / "repo"),+ "--studio-home",+ str(tmp_path / "home"),+ ])+ for key, value in overrides.items():+ setattr(args, key, value)session = module.Payload.__new__(module.Payload)session.repo_root = Path(args.repo_root)session.studio_home = Path(args.studio_home)@@ -2334,3 +2361,80 @@ def test_the_kaggle_client_is_new_enough_to_read_the_only_credential_we_have():assert pins, "no pinned kaggle client in the workflow"assert len(set(pins)) == 1, f"jobs disagree on the kaggle client: {pins}"assert packaging_version.Version(pins[0]) >= packaging_version.Version("2.2.0"), pins[0]+++def test_no_studio_assertion_is_wired_to_a_constant_branch():+ """A repo-wide version of the guard that caught five vacuous guards at once.++ Every rule in the Studio payload was, at some point, protected by a test+ that asserted "the failure message appears in the source". That is+ satisfied by `if False:` sitting above an untouched message, so disabling a+ rule outright left its test green. Five mutations survived that way in one+ sitting.++ A constant test in an `assert_*` method means a branch that can never be+ taken, or one always taken. Neither is something a check has any business+ doing, and this catches it for every assertion at once rather than one test+ at a time.+ """+ import ast++ src = (+ Path(__file__).resolve().parents[2]+ / "tests"+ / "kaggle"+ / "studio_gpu"+ / "run_studio_gpu.py"+ ).read_text(encoding = "utf-8")+ offenders = []+ for func in ast.walk(ast.parse(src)):+ if not (isinstance(func, ast.FunctionDef) and func.name.startswith("assert_")):+ continue+ for node in ast.walk(func):+ if isinstance(node, ast.If) and isinstance(node.test, ast.Constant):+ offenders.append(f"{func.name}: if {ast.unparse(node.test)}")+ assert offenders == [], f"assertions wired to a constant: {offenders}"+++def test_every_assertion_carries_its_own_wall_clock():+ """Studio is the longest payload in the kernel and had no breakdown.++ On unsloth-probe-full-concurrent-417238 `studio_test` ran 1487.5s -- the+ single largest item on the critical path -- and the report carried 19+ assertions with no timing on any of them, so "where does Studio's time go"+ could only be answered by buying another session.++ Driven rather than grepped: a rule that only checked for a `time.time()`+ call passes on a payload that computes the number and drops it.+ """+ import importlib.util+ import types++ payload = PAYLOAD_DIR / "run_studio_gpu.py"+ spec = importlib.util.spec_from_file_location("_studio_payload_timing", payload)+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)++ runner = types.SimpleNamespace(+ assertions = [],+ failures = [],+ started = module.time.time() - 5.0,+ record = None,+ )+ record = module.Payload.record.__get__(runner, module.Payload)+ module.log = lambda *a, **k: None+ record("first", True, {})+ record("second", True, {"failures": []})++ names = [a["name"] for a in runner.assertions]+ assert names == ["first", "second"]+ # The first entry measures from process start, which is the setup before+ # any assertion -- 5s here -- and is exactly the slice that would otherwise+ # be invisible.+ assert runner.assertions[0]["seconds_since_previous"] >= 5.0+ # The second measures from the first, not from the start, or every entry+ # would read as the whole run so far and the breakdown would be useless.+ assert runner.assertions[1]["seconds_since_previous"] < 1.0+ # And an absolute position, so a reader can line the report up against the+ # driver's own interval for the payload.+ assert runner.assertions[1]["at_seconds"] >= runner.assertions[0]["at_seconds"]diff --git a/tests/kaggle/test_studio_image_generation.py b/tests/kaggle/test_studio_image_generation.pynew file mode 100644index 00000000000..47c450dab77--- /dev/null+++ b/tests/kaggle/test_studio_image_generation.py@@ -0,0 +1,194 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""Image generation: the smallest run that can still be wrong in a visible way.++256x256 at 2 steps, because the claim is that the path executes rather than+that the picture is good.++"Nothing errored" is not the check, and that is the whole design. A diffusion+pipeline that fails part-way still writes a gallery record and still answers+200; a pipeline whose weights never loaded produces a FLAT frame, which is a+perfectly valid PNG. So the verdict is read off the downloaded file:++* the PNG magic, so the download endpoint is serving an image rather than a+ JSON error with a 200 on it;+* the size out of the IHDR chunk, not out of the gallery record -- the record+ repeats what was ASKED for and the file says what was MADE;+* not-one-flat-colour, on decoded extrema where PIL is available and on a+ compressed-size floor where it is not.+"""++from __future__ import annotations++import ast+import struct+import zlib+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"+SRC = PAYLOAD.read_text(encoding = "utf-8")+++def _func(name: str) -> ast.FunctionDef:+ for cls in ast.walk(ast.parse(SRC)):+ if not isinstance(cls, ast.ClassDef):+ continue+ for node in cls.body:+ if isinstance(node, ast.FunctionDef) and node.name == name:+ return node+ raise AssertionError(f"no method named {name!r}")+++def _body(name: str = "assert_image_generation") -> str:+ return ast.get_source_segment(SRC, _func(name)) or ""+++def _png(width: int, height: int, payload: bytes) -> bytes:+ def chunk(kind: bytes, data: bytes) -> bytes:+ return (+ struct.pack(">I", len(data))+ + kind+ + data+ + struct.pack(">I", zlib.crc32(kind + data) & 0xFFFFFFFF)+ )++ header = struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)+ return b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", header) + chunk(b"IDAT", payload)+++def test_the_assertion_exists_and_is_off_by_default():+ """Last priority, and it pulls a diffusion checkpoint the rest of the+ payload has no use for. A dispatch that wants it says so."""+ assert _body()+ assert "self.assert_image_generation()" in _body("execute")+ assert '"--image-generation",' in SRC+ assert "self.args.image_generation" in _body("execute")+++def test_the_run_is_the_smallest_the_schema_allows():+ body = _body()+ assert "want = 256" in body, "256 is the schema's floor for width and height"+ assert '"steps": 2,' in body+++def test_the_size_is_read_from_the_file_and_not_from_the_record():+ """The gallery record repeats what was asked for. Comparing the request+ against itself is the vacuity this rule exists against."""+ body = _body()+ assert 'int.from_bytes(png[16:20], "big")' in body+ assert 'int.from_bytes(png[20:24], "big")' in body+++def test_the_ihdr_offsets_are_right():+ """Executed against a PNG built here, because an off-by-four in a+ fixed-offset parse reads a plausible number out of the wrong bytes and+ every other rule still passes."""+ blob = _png(256, 256, zlib.compress(b"\x00" * 16))+ assert int.from_bytes(blob[16:20], "big") == 256+ assert int.from_bytes(blob[20:24], "big") == 256+ wrong = _png(512, 64, zlib.compress(b"\x00" * 16))+ assert int.from_bytes(wrong[16:20], "big") == 512+ assert int.from_bytes(wrong[20:24], "big") == 64+++def test_a_flat_image_is_a_failure():+ """A pipeline whose weights never loaded returns a uniform frame, and a+ uniform frame is a valid PNG of exactly the right size. Every other rule+ here passes on it."""+ func = _func("assert_image_generation")+ assert any(+ isinstance(n, ast.If) and ast.unparse(n.test) == "flat" for n in ast.walk(func)+ ), "nothing fails on a flat image"+ body = _body()+ assert "getextrema()" in body+ assert "flatness_source" in body, "the reader must be able to see which rule ruled"+++def test_the_png_magic_is_checked():+ body = _body()+ assert 'png.startswith(b"\\x89PNG\\r\\n\\x1a\\n")' in body+++def test_the_bytes_are_fetched_raw_rather_than_through_the_json_client():+ """`Studio.request` decodes to utf-8, which corrupts the bytes this whole+ assertion is about."""+ body = _body()+ assert "urllib.request.urlopen" in body+ assert "self.studio.get(" not in body.split("gallery/{image_id}/file")[0][-400:]+++def test_the_pipeline_is_unloaded_in_a_finally():+ """A diffusion pipeline is the largest single thing this payload puts on a+ T4. Left resident, it takes the card from whatever runs next and that+ failure lands on the wrong assertion."""+ func = _func("assert_image_generation")+ finals = "\n".join(+ ast.unparse(n) for t in ast.walk(func) if isinstance(t, ast.Try) for n in t.finalbody+ )+ assert "images/unload" in finals+++def test_a_load_or_generate_error_is_a_failure_rather_than_a_skip():+ body = _body()+ assert 'failures.append(f"images/load returned HTTP' in body+ assert "failures.append(" in body+ func = _func("assert_image_generation")+ assert any(+ isinstance(n, ast.If) and "code >= 400" in ast.unparse(n.test) for n in ast.walk(func)+ )+++def test_it_runs_while_the_server_is_still_up():+ """`assert_chat_ui` ends by clicking Stop server and asserting the port+ closes, so every request after it is refused at the socket.++ On kernel unsloth-probe-studio-full2-815a0c this assertion reported+ `URLError: Connection refused` and read as a broken image path on a server+ that had simply been shut down. Ordering, not the image pipeline.+ """+ run = _body("execute")+ image_at = run.index("self.assert_image_generation()")+ ui_at = run.index("self.assert_chat_ui()")+ assert image_at < ui_at, (+ "image generation is driven after the UI driver stops the server, so "+ "it can only ever report a connection error"+ )+++def test_the_load_is_WAITED_ON_rather_than_assumed_synchronous():+ """`images/load` answers 200 having only ACCEPTED the request.++ On kernel unsloth-probe-studio-r3-0b85d4 `load_status` was 200 and+ `generate_status` was 409 with "No diffusion model is loaded." -- an+ assertion failing on its own impatience. The wait reads `images/status`+ for `loaded`, and carries `images/load-progress` alongside so a download+ that stalls or errors is reported as that rather than as a broken+ generation.+ """+ body = _body()+ assert "/api/inference/images/status" in body+ assert "/api/inference/images/load-progress" in body+ assert 'status.get("loaded")' in body++ func = _func("assert_image_generation")+ src = ast.get_source_segment(SRC, func) or ""+ wait_at = src.index('status.get("loaded")')+ generate_at = src.index("/api/inference/images/generate")+ assert wait_at < generate_at, "the wait must come before the generate call"+++def test_a_load_that_never_finishes_fails_rather_than_generating_anyway():+ """The refusal branch. Falling through to `generate` after the deadline+ reports a 409 under the generation's name, which sends the reader after the+ wrong component -- which is exactly what happened."""+ func = _func("assert_image_generation")+ src = ast.get_source_segment(SRC, func) or ""+ marker = "never reported loaded within"+ assert marker in src+ after = src[src.index(marker) :]+ assert "return self.record" in after.split("/api/inference/images/generate")[0], (+ "the deadline branch does not return, so it generates against a model "+ "that was never loaded"+ )diff --git a/tests/kaggle/test_studio_lora_vs_base.py b/tests/kaggle/test_studio_lora_vs_base.pynew file mode 100644index 00000000000..6165b142559--- /dev/null+++ b/tests/kaggle/test_studio_lora_vs_base.py@@ -0,0 +1,174 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""The exported model against the base it came from.++`assert_gguf_export` proves a file was produced, carries the GGUF magic, loads+on the GPU and generates. Every one of those is ALSO true of an export that+silently merged nothing and shipped the base weights, which is the regression+worth catching and is invisible to file size, to the magic, and to "it+generated text".++Greedy decoding at temperature 0 makes it visible: identical weights answer+identically, so a difference is the adapter.++The determinism control is the part that makes that argument valid, and it runs+FIRST. If the same weights loaded twice do not reproduce their own answer, a+difference between two models is noise, and the assertion has to say it could+not compare rather than pass on it.++What is deliberately NOT asserted is the canary. Studio's training run is a+handful of steps, and whether that is enough to learn a specific string is a+property of the run length rather than of the export path: asserting it would+be a red about training tuning wearing an export label.+"""++from __future__ import annotations++import ast+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"+SRC = PAYLOAD.read_text(encoding = "utf-8")+++def _func(name: str) -> ast.FunctionDef:+ for cls in ast.walk(ast.parse(SRC)):+ if not isinstance(cls, ast.ClassDef):+ continue+ for node in cls.body:+ if isinstance(node, ast.FunctionDef) and node.name == name:+ return node+ raise AssertionError(f"no method named {name!r}")+++def _body(name: str = "assert_lora_vs_base") -> str:+ return ast.get_source_segment(SRC, _func(name)) or ""+++def test_the_assertion_exists_and_is_driven_from_the_run():+ assert _body()+ assert "self.assert_lora_vs_base(" in _body("execute")+++def test_it_compares_against_the_file_the_export_actually_made():+ """A path recomputed here could name a stale GGUF from an earlier run,+ which would compare the base against something this session never made."""+ body = _body("execute")+ assert 'entry["name"] == "gguf_export"' in body+ assert 'exported_gguf = entry.get("gguf")' in body+++def test_the_claim_is_that_the_two_differ():+ func = _func("assert_lora_vs_base")+ guarded = [+ n+ for n in ast.walk(func)+ if isinstance(n, ast.If) and "tuned == base_one" in ast.unparse(n.test)+ ]+ assert guarded, (+ "nothing fails when the export answers exactly as the base does, "+ "which is what a no-op merge looks like"+ )+++def test_the_determinism_control_runs_before_the_claim():+ """Ordering is the argument. A difference only means "adapter" once the+ same weights have been shown to reproduce themselves."""+ body = _body()+ control_at = body.index("base_one != base_two")+ claim_at = body.index("tuned == base_one")+ assert control_at < claim_at+++def test_a_non_reproducing_base_is_reported_as_a_failure_not_a_pass():+ """ "I could not compare" and "they matched" are opposite outcomes. Letting+ the first through as a pass is the exact shape this directory keeps being+ caught by."""+ func = _func("assert_lora_vs_base")+ for node in ast.walk(func):+ if isinstance(node, ast.If) and "base_one != base_two" in ast.unparse(node.test):+ appends = [+ n+ for n in ast.walk(node)+ if isinstance(n, ast.Call)+ and isinstance(n.func, ast.Attribute)+ and n.func.attr == "append"+ ]+ assert appends, "a base that will not reproduce itself must fail"+ break+ else:+ raise AssertionError("there is no determinism control at all")+++def test_the_canary_is_recorded_and_never_asserted():+ """It is a property of how many steps the training ran, not of the export+ path. Failing on it would put a training-tuning red under an export name."""+ func = _func("assert_lora_vs_base")+ for node in ast.walk(func):+ if isinstance(node, ast.If) and "CANARY" in ast.unparse(node.test):+ raise AssertionError(+ "the verdict branches on the canary, which makes this red "+ "whenever the training run is too short to learn it"+ )+ assert "canary_in_exported" in _body()+++def test_a_missing_export_is_a_failure_rather_than_a_skip():+ body = _body()+ assert "if not gguf:" in body+ assert 'return self.record("lora_vs_base", False, detail)' in body+++def test_both_models_are_driven_with_the_same_prompt():+ """Two different questions would differ for reasons that have nothing to+ do with the adapter."""+ func = _func("assert_lora_vs_base")+ prompts = [+ n+ for n in ast.walk(func)+ if isinstance(n, ast.Assign)+ and any(isinstance(t, ast.Name) and t.id == "prompt" for t in n.targets)+ ]+ assert len(prompts) == 1, "the prompt must be built once and reused"+ assert (+ _body().count("self.chat(prompt)") == 1+ ), "one call site, reached by every arm, or the arms can drift apart"+++def test_the_exported_gguf_picked_is_a_model_and_not_an_mmproj_sidecar():+ """Driven through the REAL `newest_gguf`, because this is a selection bug+ and a rule fed a path proves nothing about the selector.++ A vision export writes two files, and the projector is often the newer.+ Handing `Qwen3.5-2B.F16-mmproj.gguf` to llama.cpp as a model is not an+ error: the server starts, reports gpu_layers=-1, offloads nothing and still+ returns text. On kernel unsloth-probe-studio-full2-815a0c that failed both+ the export assertion and this one, as a GPU fallback that never happened.+ """+ import sys as _sys # noqa: PLC0415+ import tempfile # noqa: PLC0415+ import time as _time # noqa: PLC0415+ from pathlib import Path as _Path # noqa: PLC0415++ _sys.path.insert(0, str(ROOT / "tests" / "kaggle" / "studio_gpu"))+ from studio_client import newest_gguf # noqa: PLC0415++ with tempfile.TemporaryDirectory() as tmp:+ root = _Path(tmp)+ model = root / "Qwen3.5-2B.Q8_0.gguf"+ model.write_bytes(b"GGUF")+ _time.sleep(0.01)+ # Written LAST, so "newest" alone would pick it.+ sidecar = root / "Qwen3.5-2B.F16-mmproj.gguf"+ sidecar.write_bytes(b"GGUF")++ assert newest_gguf(root) == model++ # And with only a sidecar present, the answer is None rather than the+ # sidecar: no model was exported, and saying so is the point.+ sidecar_only = root / "sub"+ sidecar_only.mkdir()+ (sidecar_only / "x.F16-mmproj.gguf").write_bytes(b"GGUF")+ assert newest_gguf(sidecar_only) is Nonediff --git a/tests/kaggle/test_studio_mtp_engages.py b/tests/kaggle/test_studio_mtp_engages.pynew file mode 100644index 00000000000..8be55969d9d--- /dev/null+++ b/tests/kaggle/test_studio_mtp_engages.py@@ -0,0 +1,96 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""Selecting the MTP repo is a REQUEST; the drafter engaging is the result.++`--chat-model unsloth/Qwen3.5-2B-MTP-GGUF` was chosen because multi-token+prediction is a distinct serving path in llama.cpp. But when the companion is+missing, the binary was built without MTP, or the drafter is downgraded for+VRAM, the main GGUF still loads and still generates -- so every inference+assertion stays green while the path the repo was chosen for never ran. That is+the shape this directory keeps being caught by: nothing red, and a claim nobody+made.++`spec_drafter_kind` is what settles it, and `spec_fallback_reason` names WHICH+of those happened, since "llama.cpp has no MTP" and "the drafter was downgraded+for VRAM" are different findings and only one is about this leg.+"""++from __future__ import annotations++import ast+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"+SRC = PAYLOAD.read_text(encoding = "utf-8")+TREE = ast.parse(SRC)+++def _func(name: str) -> ast.FunctionDef:+ for node in ast.walk(TREE):+ if isinstance(node, ast.FunctionDef) and node.name == name:+ return node+ raise AssertionError(f"{name} is gone")+++def test_the_default_chat_model_is_the_mtp_repo():+ """If this stops being an MTP repo the assertion below becomes vacuous, so+ it fails here rather than passing silently."""+ assert "unsloth/Qwen3.5-2B-MTP-GGUF" in SRC+++def test_the_status_fields_are_read_from_the_load():+ body = ast.unparse(_func("load_model"))+ for field in ("spec_drafter_kind", "spec_fallback_reason", "llama_cpp_supports_mtp"):+ assert field in body, f"{field} is never read, so the drafter cannot be checked"+++def test_a_drafter_that_never_engaged_is_a_failure():+ """The claim itself. Without this, an MTP repo whose drafter fell back+ serves ordinary decoding and reports success."""+ func = _func("assert_gpu_inference")+ body = ast.unparse(func)+ assert "drafter_kind" in body, "the drafter kind is never compared"+ guards = [+ ast.unparse(node.test)+ for node in ast.walk(func)+ if isinstance(node, ast.If) and "drafter_kind" in ast.unparse(node.test)+ ]+ assert guards, "spec_drafter_kind appears but gates nothing"+ assert any(+ "'mtp'" in guard or '"mtp"' in guard for guard in guards+ ), "the drafter kind is read but never required to be mtp"+ appends = [+ ast.unparse(node)+ for node in ast.walk(func)+ if isinstance(node, ast.Call)+ and "failures" in ast.unparse(node)+ and "append" in ast.unparse(node)+ and "drafter" in ast.unparse(node)+ ]+ assert appends, "a fallback is noticed but not recorded as a failure"+++def test_the_check_is_scoped_to_an_mtp_model():+ """A leg pointed at a plain GGUF must not go red for not doing MTP: that is+ the correct behaviour there, and a rule that cannot pass is deleted."""+ func = _func("assert_gpu_inference")+ guards = [+ ast.unparse(node.test)+ for node in ast.walk(func)+ if isinstance(node, ast.If) and "drafter_kind" in ast.unparse(node.test)+ ]+ assert any("MTP" in guard and "chat_model" in guard for guard in guards), (+ "the MTP requirement is unconditional, so a plain-GGUF run fails on "+ "behaviour that is correct for it"+ )+++def test_the_reason_is_reported_and_not_just_the_verdict():+ """'MTP is off' sends the reader nowhere. The reason distinguishes a build+ without MTP from a VRAM downgrade, and only one of those is actionable."""+ func = _func("assert_gpu_inference")+ body = ast.unparse(func)+ assert "fallback_reason" in body+ assert "supports_mtp" in bodydiff --git a/tests/kaggle/test_studio_password_path.py b/tests/kaggle/test_studio_password_path.pynew file mode 100644index 00000000000..273bc8f623e--- /dev/null+++ b/tests/kaggle/test_studio_password_path.py@@ -0,0 +1,107 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""Starting Studio headless with --password, and the way that check goes vacuous.++`unsloth studio --password X` sets the INITIAL admin password when none is set+yet, which is exactly the state a scripted server is in. Without it the only way+in is the bootstrap password Studio seeds into a file and prints to its own log,+and a deployment that has to read a log to log in is not one anybody scripts+twice.++**The vacuity to avoid is a fallback.** If logging in with the passed password+failed and the payload then tried the bootstrap, the run would go green while+proving `--password` does nothing at all. There is deliberately no fallback, and+that is asserted from the source here.++**The second risk is a leaked credential.** The password reaches a process+command line, Studio's startup banner, `studio.log`, and the evidence bundle+that travels home. It is generated per run and registered as a secret before+anything can log it, so every log leaving the machine is scrubbed.+"""++from __future__ import annotations++import ast+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"+SRC = PAYLOAD.read_text(encoding = "utf-8")+++def test_the_password_reaches_the_studio_command():+ assert 'cmd += ["--password", self.args.studio_password]' in SRC+++def test_login_uses_the_password_that_was_passed():+ assert "self.studio.login(self.args.studio_password)" in SRC+++def test_there_is_no_fallback_to_the_bootstrap_password():+ """The whole assertion. If --password were ignored, Studio would seed a+ bootstrap password instead and the login would fail; a fallback would then+ quietly succeed and report a pass for a flag that did nothing."""+ tree = ast.parse(SRC)+ func = next(+ node+ for node in ast.walk(tree)+ if isinstance(node, ast.FunctionDef) and node.name == "authenticate"+ )+ # Find the `if self.args.studio_password:` branch and assert it RETURNS+ # rather than falling through into the bootstrap path below it.+ branch = next(+ node+ for node in func.body+ if isinstance(node, ast.If) and "studio_password" in ast.dump(node.test)+ )+ # The LAST statement of the branch, unconditionally, and not merely "a+ # Return somewhere inside". `if not failures: return ...` contains a Return+ # and still falls through on the failure path -- which is the single case+ # this guard exists for. That mutation survived the first version of this+ # assertion, so the rule is now about the branch always returning.+ assert isinstance(branch.body[-1], ast.Return), (+ "the --password branch must END in an unconditional return; a "+ "conditional one falls through to the bootstrap path on failure and "+ "turns a flag that did nothing into a pass"+ )+ branch_src = ast.get_source_segment(SRC, branch) or ""+ assert "remember_bootstrap" not in branch_src, "the --password branch reads the bootstrap"+++def test_the_generated_password_is_registered_as_a_secret_before_use():+ """Registered in __init__, which runs before the server starts and before+ anything writes a log. Registering it later would scrub the logs written+ after that point and not the banner."""+ tree = ast.parse(SRC)+ init = next(+ node+ for node in ast.walk(tree)+ if isinstance(node, ast.FunctionDef) and node.name == "__init__"+ )+ body = ast.get_source_segment(SRC, init) or ""+ assert "self.secrets.add(self.args.studio_password)" in body+ assert "secrets_module.token_urlsafe" in body, "auto must mint a fresh value per run"+++def test_no_constant_password_is_committed():+ """A constant in a repo is a credential whether or not it is reachable."""+ tree = ast.parse(SRC)+ for node in ast.walk(tree):+ if isinstance(node, ast.Call) and getattr(node.func, "attr", "") == "add_argument":+ names = [a.value for a in node.args if isinstance(a, ast.Constant)]+ if "--studio-password" in names:+ for kw in node.keywords:+ if kw.arg == "default":+ assert (+ kw.value.value == ""+ ), f"a default password is committed: {kw.value.value!r}"+ break+ else:+ raise AssertionError("--studio-password is not declared at all")+++def test_the_default_keeps_the_previous_bootstrap_behaviour():+ """Empty means "use the bootstrap", so a caller that does not opt in is+ unaffected and the existing Studio leg does not change shape."""+ assert '"--studio-password",\n default = "",' in SRCdiff --git a/tests/kaggle/test_studio_server_flags.py b/tests/kaggle/test_studio_server_flags.pynew file mode 100644index 00000000000..785a56b0b87--- /dev/null+++ b/tests/kaggle/test_studio_server_flags.py@@ -0,0 +1,283 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""llama.cpp server flags: a quantized KV cache, a two-card split, a pinned ctx.++The vacuity here is specific and easy to write by accident: **asserting that+the load succeeded**. llama-server starts happily when a flag it does not like+is dropped, so a load that ignored `cache_type_kv` entirely, fell back to one+card, and ran at the model's default context is indistinguishable from a+correct one by return code alone.++Studio's status separates the request from what is in force, which is what+makes a real check possible. So the rules are about the APPLIED values, and a+downgrade is allowed only when Studio says why -- a model whose cache layout+cannot be quantized is entitled to refuse, and silence is not.+"""++from __future__ import annotations++import ast+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"+SRC = PAYLOAD.read_text(encoding = "utf-8")+++def _func(name: str = "assert_server_flags") -> ast.FunctionDef:+ tree = ast.parse(SRC)+ return next(n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == name)+++def _body() -> str:+ tree = ast.parse(SRC)+ func = next(+ n+ for n in ast.walk(tree)+ if isinstance(n, ast.FunctionDef) and n.name == "assert_server_flags"+ )+ return ast.get_source_segment(SRC, func) or ""+++def test_the_load_requests_all_three_flags():+ body = _body()+ assert '"cache_type_kv": "q8_0"' in body+ # Sized to the VISIBLE cards rather than hardcoded to two.+ # --studio-concurrent pins this half to one card so it can share with a+ # training leg, and [1.0, 1.0] against a one-card server asks llama.cpp to+ # split across a device that is not there: what comes back is a failure+ # about the load rather than about the flag.+ assert '"tensor_split": [1.0] * max(1, len(cards))' in body+ assert "cards = gpu_inventory()" in body+ assert '"max_seq_length": self.args.studio_ctx' in body+++def test_a_one_card_run_says_the_two_card_split_was_not_exercised():+ """The coverage this shares away must be STATED, not silently dropped.++ Under --studio-concurrent the split is over one device, which is not the+ flag the brief asks about. A check that keeps its name while testing less+ is the failure this directory keeps being caught by, so the report carries+ `tensor_split_over_two_cards` and a note naming what was and was not+ covered.+ """+ body = _body()+ assert 'detail["tensor_split_over_two_cards"] = len(cards) >= 2' in body+ assert "was NOT exercised" in body+ # And it is a RECORD, not a failure: sharing is a scheduling decision, so a+ # one-card run is not a defect and must not go red for being one.+ func = _func("assert_server_flags")+ for node in ast.walk(func):+ if isinstance(node, ast.If) and "cards" in ast.unparse(node.test):+ appended = [+ inner+ for inner in ast.walk(node)+ if isinstance(inner, ast.Call)+ and isinstance(inner.func, ast.Attribute)+ and inner.func.attr == "append"+ ]+ assert not appended, (+ "a one-card run fails rather than records, so every "+ "--studio-concurrent run would go red by design"+ )+++def test_the_check_reads_the_applied_values_not_the_request():+ """The whole point. A check on the request is a check on a dict this file+ just built."""+ body = _body()+ assert '"/api/inference/status"' in body+ assert 'applied.get("cache_type_kv")' in body+ assert 'applied.get("context_length")' in body+++def test_no_branch_in_the_check_is_wired_to_a_constant():+ """The guard that caught five vacuous guards, including four of my own.++ Every rule in this file was first written as "the failure message appears+ in the source". That is satisfied by `if False:` above an untouched+ message, so disabling a rule outright left the test green -- the exact+ "assertion satisfied by its own surrounding text" failure this repo has+ recorded before.++ A constant test means a branch that can never be taken (or always is), and+ no rule here has any business being either.+ """+ tree = ast.parse(_body())+ constants = [+ ast.unparse(node.test)+ for node in ast.walk(tree)+ if isinstance(node, ast.If) and isinstance(node.test, ast.Constant)+ ]+ assert constants == [], f"branches wired to a constant: {constants}"+++def test_the_cache_rules_append_a_failure_on_both_paths():+ """Both the missing-value path and the silent-downgrade path must be able+ to fail, and each is reached through its own branch."""+ tree = ast.parse(_body())+ messages = [+ ast.unparse(node)+ for node in ast.walk(tree)+ if isinstance(node, ast.Call) and getattr(node.func, "attr", "") == "append"+ ]+ joined = " ".join(messages)+ assert "no cache_type_kv at all" in joined+ assert "silent downgrade" in joined.lower()+++def test_the_context_pin_is_checked_against_the_requested_value():+ """llama-server admits a prompt on n_ctx alone, so a server at the model+ default behaves differently from one at 2048 and the difference does not+ show in a chat response."""+ body = _body()+ assert "int(ctx) > self.args.studio_ctx" in body+++def test_gpu_residency_is_confirmed_after_the_split():+ """A tensor_split that fell back to CPU reports a healthy server and proves+ nothing about either card. Checked structurally: the threshold comparison+ must exist, not merely the message about it."""+ tree = ast.parse(_body())+ compares = [ast.unparse(node) for node in ast.walk(tree) if isinstance(node, ast.Compare)]+ assert any(+ "used" in c and "200" in c for c in compares+ ), f"no residency threshold comparison: {compares}"+ assert "nvidia_used_mib()" in _body()+++def test_it_is_skipped_rather_than_passed_when_the_model_is_not_on_the_gpu():+ """Structural: the skip branch must record FALSE. Recording True would turn+ "we could not test this" into "this passed"."""+ tree = ast.parse(SRC)+ records = [+ node+ for node in ast.walk(tree)+ if isinstance(node, ast.Call)+ and getattr(node.func, "attr", "") == "record"+ and node.args+ and isinstance(node.args[0], ast.Constant)+ and node.args[0].value == "server_flags"+ ]+ skips = [n for n in records if len(n.args) > 1 and "skipped" in ast.unparse(n)]+ assert skips, "no skip path records server_flags"+ for node in skips:+ assert (+ node.args[1].value is False+ ), "the skip path records a PASS, so an untested flag reads as a working one"+++def test_it_runs_between_gpu_inference_and_training():+ """It reloads the chat model with different flags. Earlier would change the+ model the inference checks measured; later would put a reload between the+ adapter and the export."""+ infer_at = SRC.index("gpu_ok = self.assert_gpu_inference()")+ flags_at = SRC.index("self.assert_server_flags()")+ train_at = SRC.index("trained = self.assert_training()")+ assert infer_at < flags_at < train_at++ # And it is guarded by gpu_ok rather than by a constant, or the call sits+ # in the right place and never runs.+ tree = ast.parse(SRC)+ calls = [+ node+ for node in ast.walk(tree)+ if isinstance(node, ast.If)+ and "assert_server_flags" in ast.unparse(node)+ and not isinstance(node, ast.Module)+ ]+ assert calls, "the call is not inside a branch at all"+ assert any(ast.unparse(n.test) == "gpu_ok" for n in calls), (+ f"the call is gated on something other than gpu_ok: "+ f"{[ast.unparse(n.test) for n in calls]}"+ )+++def test_the_context_default_is_the_one_the_brief_asks_for():+ assert '"--studio-ctx",' in SRC+ assert "default = 2048," in SRC+++def _payload_module():+ """The real payload, imported by path so the rules DRIVE it.++ Every rule above this point reads the source with `ast`, which is the right+ instrument for "does the branch exist" and the wrong one for "does it answer+ correctly". The bug below was invisible to all of them: the code was+ exactly as written and the value it produced was false.+ """+ import importlib.util++ spec = importlib.util.spec_from_file_location("_studio_payload_flags", PAYLOAD)+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return module+++def _with_smi(module, monkeypatch, rows: str, visible: str | None):+ class _Proc:+ returncode = 0+ stdout = rows++ monkeypatch.setattr(module, "run", lambda *a, **k: _Proc())+ if visible is None:+ monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising = False)+ else:+ monkeypatch.setenv("CUDA_VISIBLE_DEVICES", visible)+++TWO_ROWS = "Tesla T4, 15360 MiB, 7.5\nTesla T4, 15360 MiB, 7.5\n"+++def test_a_pinned_payload_does_not_see_both_cards(monkeypatch):+ """The exact reading from unsloth-probe-full-concurrent-417238.++ nvidia-smi lists two T4s. build_kernel.py:835 pinned this payload to card 0+ with CUDA_VISIBLE_DEVICES. The report recorded `cards_visible: 2` and+ `tensor_split_over_two_cards: True` and sent `tensor_split: [1.0, 1.0]` to a+ one-card server, which loaded anyway -- so the assertion passed green while+ asking llama.cpp to split across a device that was not there.+ """+ module = _payload_module()+ _with_smi(module, monkeypatch, TWO_ROWS, "0")+ assert len(module.gpu_inventory()) == 1+++def test_an_unpinned_payload_still_sees_every_card(monkeypatch):+ """The other direction, and it is the one the two-card claim depends on: a+ leg that is deliberately NOT pinned must still report both cards, or real+ multi-GPU coverage would be reported as single-card."""+ module = _payload_module()+ _with_smi(module, monkeypatch, TWO_ROWS, None)+ assert len(module.gpu_inventory()) == 2+ _with_smi(module, monkeypatch, TWO_ROWS, "0,1")+ assert len(module.gpu_inventory()) == 2+++def test_an_empty_setting_means_no_cards_rather_than_all_of_them(monkeypatch):+ """CUDA_VISIBLE_DEVICES="" is a deliberate "no GPU" and is not the same as+ unset. Reading it as unset would report a full inventory on a session that+ has no card at all, which preflight exists to catch."""+ module = _payload_module()+ _with_smi(module, monkeypatch, TWO_ROWS, "")+ assert module.gpu_inventory() == []+++def test_a_uuid_selection_still_counts_the_cards(monkeypatch):+ """CUDA_VISIBLE_DEVICES may name GPU-<uuid> rather than an index. The+ description cannot be recovered from an nvidia-smi row, but the COUNT is+ what tensor_split is sized from, so it must stay right."""+ module = _payload_module()+ _with_smi(module, monkeypatch, TWO_ROWS, "GPU-abcdef12")+ assert len(module.gpu_inventory()) == 1+++def test_the_split_is_sized_from_the_visible_cards_and_not_from_nvidia_smi():+ """The link between the fix and the flag. `tensor_split` is built from+ `gpu_inventory()`, so a rule that only tested the helper would leave the+ request free to be sized from anything."""+ body = _body()+ assert "cards = gpu_inventory()" in body+ assert "[1.0] * max(1, len(cards))" in body+ assert 'detail["tensor_split_over_two_cards"] = len(cards) >= 2' in bodydiff --git a/tests/kaggle/test_studio_tabs.py b/tests/kaggle/test_studio_tabs.pynew file mode 100644index 00000000000..2dd7e6e985a--- /dev/null+++ b/tests/kaggle/test_studio_tabs.py@@ -0,0 +1,139 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""Every Studio tab's backing endpoint, checked against the routes that exist.++The rest of this payload touches inference, training and export. A tab whose+router raised on import is invisible to all of it: the route is simply never+mounted, and the tab renders an error the moment a user opens it.++This is a smoke and says so. What makes it more than a list of strings is the+rule below that every path in `TAB_ENDPOINTS` resolves to a route actually+declared in `studio/backend/routes`. A typo'd path 404s exactly like a missing+router, so without that rule the payload could go red for the wrong reason --+or, worse, a path could be quietly "fixed" to something that always answers.++One endpoint choice is load-bearing: `/api/data-recipe/jobs/current` 404s by+design when no job is running, so using it would be red on correct behaviour.+"""++from __future__ import annotations++import ast+import re+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"+SRC = PAYLOAD.read_text(encoding = "utf-8")+ROUTES = ROOT / "studio" / "backend" / "routes"+MAIN = ROOT / "studio" / "backend" / "main.py"+++def _cls() -> ast.ClassDef:+ return next(n for n in ast.walk(ast.parse(SRC)) if isinstance(n, ast.ClassDef))+++def _endpoints() -> list[tuple[str, str]]:+ for node in _cls().body:+ if isinstance(node, ast.Assign) and any(+ isinstance(t, ast.Name) and t.id == "TAB_ENDPOINTS" for t in node.targets+ ):+ return [+ (e.elts[0].value, e.elts[1].value)+ for e in node.value.elts+ if isinstance(e, ast.Tuple)+ ]+ raise AssertionError("TAB_ENDPOINTS is gone")+++def _func(name: str) -> ast.FunctionDef:+ for node in _cls().body:+ if isinstance(node, ast.FunctionDef) and node.name == name:+ return node+ raise AssertionError(f"no method named {name!r}")+++def _body(name: str = "assert_tabs") -> str:+ return ast.get_source_segment(SRC, _func(name)) or ""+++def _mounted_prefixes() -> set[str]:+ return set(+ re.findall(r'include_router\([^)]*prefix\s*=\s*"([^"]+)"', MAIN.read_text(encoding = "utf-8"))+ )+++def _declared_get_paths() -> set[str]:+ paths: set[str] = set()+ for path in ROUTES.rglob("*.py"):+ paths.update(re.findall(r'@\w*router\.get\(\s*"([^"]+)"', path.read_text(encoding = "utf-8")))+ return paths+++def test_the_assertion_exists_and_is_driven_from_the_run():+ assert _body()+ assert "self.assert_tabs()" in _body("execute")+++def test_every_endpoint_resolves_to_a_route_that_exists():+ """The rule that turns a list of strings into a check. A typo'd path 404s+ exactly like a missing router, so the payload would go red naming a tab+ that is perfectly fine."""+ prefixes = _mounted_prefixes()+ declared = _declared_get_paths()+ unresolved = []+ for name, path in _endpoints():+ for prefix in prefixes:+ if path.startswith(prefix) and (path[len(prefix) :] or "/") in declared:+ break+ else:+ unresolved.append(f"{name}: {path}")+ assert unresolved == [], f"these paths match no declared GET route: {unresolved}"+++def test_the_four_tabs_the_directive_names_are_all_covered():+ names = {name for name, _ in _endpoints()}+ for wanted in ("data_designer", "image_creation", "video_creation", "image_training"):+ assert any(n.startswith(wanted) for n in names), f"{wanted} is not covered"+++def test_it_does_not_use_an_endpoint_that_404s_by_design():+ """`/jobs/current` raises 404 when no job is running. Using it would make+ this red on correct behaviour, which is how a check gets switched off."""+ assert all(path != "/api/data-recipe/jobs/current" for _, path in _endpoints())+++def test_a_404_is_distinguished_from_other_failures():+ """A 500 is a live route with a broken handler; a 404 is a route that does+ not exist. They send a reader to different places."""+ func = _func("assert_tabs")+ tests = [ast.unparse(n.test) for n in ast.walk(func) if isinstance(n, ast.If)]+ assert any(t == "code == 404" for t in tests)+ assert any("code >= 400" in t for t in tests)+++def test_a_200_that_is_not_a_json_object_fails():+ """Most of these are declared with a response_model, so a bare string or a+ null means something is standing in for the real handler."""+ func = _func("assert_tabs")+ assert any(+ "isinstance(body, (dict, list))" in ast.unparse(n.test)+ for n in ast.walk(func)+ if isinstance(n, ast.If)+ )+++def test_an_endpoint_that_raises_is_a_failure_rather_than_a_skip():+ func = _func("assert_tabs")+ handlers = [n for n in ast.walk(func) if isinstance(n, ast.ExceptHandler)]+ assert handlers, "a transport error must not end the loop silently"+ joined = "\n".join(ast.unparse(h) for h in handlers)+ assert "failures.append" in joined+++def test_it_runs_before_the_long_training_phase():+ """These need only a logged-in session. Behind a 20-minute training run, a+ training failure hides whether the tabs exist at all."""+ body = _body("execute")+ assert body.index("self.assert_tabs()") < body.index("self.assert_training()")diff --git a/tests/kaggle/test_studio_web_search.py b/tests/kaggle/test_studio_web_search.pynew file mode 100644index 00000000000..6e285d0c903--- /dev/null+++ b/tests/kaggle/test_studio_web_search.py@@ -0,0 +1,201 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""The web_search tool must be executed, and the log is the only witness.++`assert_code_execution` reads its verdict off the filesystem; a web search+leaves nothing there. What it does leave is Studio's own+`execute_tool: name=...` line, emitted from INSIDE `execute_tool`, so it is+written by execution rather than by selection. A loop that hands the model the+schema and never runs the call produces no such line.++Two rules stop the check drifting into vacuity in opposite directions.++It must count only what THIS request wrote. The payload runs several+tool-driven assertions against one long-lived server, and a whole-file grep+would let `assert_code_execution`'s call satisfy this one.++And it must NOT fail on an empty result set. `_web_search` fans out through+ddgs with no API key, so a provider rate-limiting a Kaggle egress IP is a fact+about the day rather than a Studio defect, and failing on it would put a red in+front of every PR that no reader could act on.+"""++from __future__ import annotations++import ast+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "studio_gpu" / "run_studio_gpu.py"+SRC = PAYLOAD.read_text(encoding = "utf-8")+++def _func(name: str) -> ast.FunctionDef:+ for cls in ast.walk(ast.parse(SRC)):+ if not isinstance(cls, ast.ClassDef):+ continue+ for node in cls.body:+ if isinstance(node, ast.FunctionDef) and node.name == name:+ return node+ raise AssertionError(f"no method named {name!r}")+++def _body(name: str = "assert_web_search") -> str:+ return ast.get_source_segment(SRC, _func(name)) or ""+++def test_the_assertion_exists_and_is_driven_from_the_run():+ assert _body()+ assert "self.assert_web_search()" in _body("execute")+++def test_only_the_web_search_tool_is_offered():+ """With `enabled_tools` omitted the loop may reach for python instead, and+ a run that executed python would satisfy a looser log check while saying+ nothing about search."""+ body = _body()+ assert "enable_tools = True" in body+ assert 'enabled_tools = ["web_search"]' in body+++def test_the_evidence_is_the_execution_line_and_not_a_selection_one():+ body = _body()+ assert 'marker = "execute_tool: name=web_search"' in body, (+ "the line must be the one execute_tool writes, because a line written "+ "where the tool is CHOSEN is emitted whether or not it then runs"+ )+++def test_it_counts_only_what_this_request_wrote():+ """The payload drives several tool assertions against one server. A grep+ over the whole log would let an earlier assertion's tool call stand in for+ this one, which is a green tick for a search that never happened."""+ body = _body()+ assert "before = self.server_log.read_text" in body+ # Whitespace-insensitive: the repo's formatter rewrites this slice to+ # `after[len(before) :]`, and a guard that matched the unformatted spelling+ # went red on a reformat rather than on a regression.+ assert "fresh=after[len(before):]" in "".join(body.split())+ assert "fresh.count(marker)" in body, "counted over the fresh slice, not the file"+++def test_an_empty_result_set_is_reported_and_not_failed():+ """Deliberately narrow. ddgs runs with no API key, so a provider throttling+ a Kaggle egress IP is a fact about the day; failing on it would be a red+ nobody can act on. The execution is the claim; the results are context."""+ func = _func("assert_web_search")+ for node in ast.walk(func):+ if not isinstance(node, ast.If):+ continue+ test = ast.unparse(node.test)+ appends = [+ n+ for n in ast.walk(node)+ if isinstance(n, ast.Call)+ and isinstance(n.func, ast.Attribute)+ and n.func.attr == "append"+ ]+ if appends and ("results" in test or "reply" in test):+ raise AssertionError(+ f"failing on {test!r} makes this red whenever the search "+ f"provider is having a bad day"+ )+++def test_the_failure_fires_when_nothing_executed():+ func = _func("assert_web_search")+ # Only the branches that DECIDE the verdict. The loop also breaks early on+ # a positive count, which is a control-flow test rather than a rule, and+ # reading it as one made this guard red on a correct body.+ guarded = [+ node+ for node in ast.walk(func)+ if isinstance(node, ast.If)+ and "executions" in ast.unparse(node.test)+ and any(+ isinstance(inner, ast.Call)+ and isinstance(inner.func, ast.Attribute)+ and inner.func.attr == "append"+ for inner in ast.walk(node)+ )+ ]+ assert guarded, "nothing that decides the verdict depends on the tool having run"+ assert all(+ isinstance(n.test, ast.UnaryOp) and isinstance(n.test.op, ast.Not) for n in guarded+ ), "the failure must fire on ZERO executions"+++def test_the_cpu_fallback_records_the_assertion_rather_than_omitting_it():+ assert '"web_search",' in _body("execute")+++def test_the_search_tool_call_is_FORCED_rather_than_hoped_for():+ """Otherwise this measures a model's judgement, not Studio's plumbing.++ On kernel unsloth-probe-studio-full2-815a0c the 2B model answered "The+ current version of the Linux kernel is 6.10" straight from parametric+ knowledge, never emitted a call, and the assertion reported that Studio had+ offered web_search and not run it. Studio had done nothing wrong.++ `assert_tool_calling` already forces its own tool the same way. The claim+ here is that Studio EXECUTES the call, and the `execute_tool` log check+ still decides that, so forcing the call narrows the assertion onto the+ thing it is about rather than weakening it.+ """+ body = _body()+ # BY NAME, not the bare "required": that was tried on kernel+ # unsloth-probe-studio-r3-0b85d4 and the model still answered from+ # parametric knowledge with executions 0.+ assert '"function": {"name": "web_search"}' in body+ assert 'tool_choice = "required"' not in body+++def test_both_tool_selections_are_tried_before_the_verdict():+ """One attempt cannot tell a selection bug from a model that will not+ search.++ `enabled_tools = ["web_search"]` is one name out of ALL_TOOLS, and+ `routes/inference.py` also reads a request naming only hosted-tool names as+ a provider-hosted ask. Omitting `enabled_tools` selects every local tool,+ which is a different path through the same loop. Reporting "the loop+ offered web_search and never ran it" off the first alone was a guess: no+ evidence in that run showed the tool had been offered at all.+ """+ body = _body()+ assert '("named", {"enabled_tools": ["web_search"]})' in body+ assert '("all_local_tools", {})' in body+ assert '"any_tool_executions"' in body, (+ "without a count of ANY tool execution, a loop that ran and chose "+ "something else is indistinguishable from a loop that never ran"+ )+++def test_the_second_attempt_is_skipped_once_one_succeeds():+ """A passing first attempt must not spend a second inference on the same+ claim; the loop breaks on a positive count."""+ func = _func("assert_web_search")+ src = ast.get_source_segment(SRC, func) or ""+ assert 'if record["executions"]:' in src+ assert "break" in src+++def test_the_verdict_counts_web_search_and_not_any_tool():+ """Mutation found this: summing `any_tool_executions` instead passes on the+ python tool being run, which is a different assertion in this same payload.+ The wider count is diagnostic context, never the rule."""+ func = _func("assert_web_search")+ verdict = next(+ node+ for node in ast.walk(func)+ if isinstance(node, ast.Assign)+ and any(+ isinstance(t, ast.Subscript) and ast.unparse(t) == "detail['executions']"+ for t in node.targets+ )+ )+ source = ast.unparse(verdict.value)+ assert "'executions'" in source+ assert (+ "any_tool_executions" not in source+ ), "the verdict counts any tool at all, so the python tool satisfies the web-search claim"diff --git a/tests/kaggle/test_t4_ci_transport.py b/tests/kaggle/test_t4_ci_transport.pyindex 7f824709b31..62c71a9531b 100644--- a/tests/kaggle/test_t4_ci_transport.py+++ b/tests/kaggle/test_t4_ci_transport.py@@ -14,6 +14,7 @@ from __future__ import annotationsimport jsonimport os+import reimport subprocessimport sysimport tempfile@@ -23,6 +24,7 @@ import typesfrom pathlib import Pathimport pytest+import yamlREPO_ROOT = Path(__file__).resolve().parents[2]SMOKE_DIR = REPO_ROOT / "tests" / "kaggle" / "t4_smoke"@@ -34,7 +36,7 @@ sys.path.insert(0, str(CI_DIR))import build_kernel # noqa: E402import gate # noqa: E402import launch # noqa: E402-from legs import LEGS # noqa: E402+from legs import KERNELS, LEGS # noqa: E402# ------------------------------------------------------------------ driver@@ -57,6 +59,19 @@ class _Stub:self.gpus = gpusself.venv_ok = venv_okself.papermill: list[dict] = []+ # Every `pip install --target ...` the driver issued, which is the only+ # place an overlay's CONTENTS are decided. Recorded as the full command+ # so a guard can ask what was installed, not merely that something was.+ self.overlay_installs: list[list[str]] = []+ # What the resolver is pretended to have found. Deliberately mixed: one+ # ordinary pure-Python distribution and one native one, so the driver's+ # deny-list has something real to reject. A stub closure of only safe+ # packages would let a driver with no deny-list at all pass.+ self.resolver_closure = [+ ("transformers", "4.57.6"),+ ("trl", "0.22.2"),+ ("torch", "2.99.0"),+ ]self.TimeoutExpired = subprocess.TimeoutExpiredself.CalledProcessError = subprocess.CalledProcessErrorself.STDOUT = subprocess.STDOUT@@ -70,6 +85,30 @@ class _Stub:return types.SimpleNamespace(returncode = 0, stdout = out, stderr = "")if cmd[0] == "which":return types.SimpleNamespace(returncode = 0, stdout = "/usr/bin/uv\n", stderr = "")+ if "--report" in cmd:+ # `pip install --dry-run --report FILE` writes the resolved closure+ # to FILE and prints nothing useful, so a stub that only returns a+ # returncode leaves the driver with an empty manifest -- which it+ # handles by installing nothing, and the overlay guard would then+ # pass while proving the overlay never happened.+ report = Path(cmd[cmd.index("--report") + 1])+ report.parent.mkdir(parents = True, exist_ok = True)+ report.write_text(+ json.dumps(+ {+ "install": [+ {"metadata": {"name": n, "version": v}}+ for n, v in self.resolver_closure+ ]+ }+ ),+ encoding = "utf-8",+ )+ return types.SimpleNamespace(returncode = 0, stdout = "", stderr = "")+ if "--target" in cmd:+ self.overlay_installs.append(list(cmd))+ Path(cmd[cmd.index("--target") + 1]).mkdir(parents = True, exist_ok = True)+ return types.SimpleNamespace(returncode = 0, stdout = "", stderr = "")if "papermill" in cmd:env = kw.get("env") or {}self.papermill.append(@@ -78,6 +117,10 @@ class _Stub:"cuda": env.get("CUDA_VISIBLE_DEVICES"),"kernel": cmd[cmd.index("-k") + 1],"compile_location": env.get("UNSLOTH_COMPILE_LOCATION"),+ # The whole env, because the caches this file now asserts+ # on are several variables and a recorder that names them+ # one at a time goes stale the moment another is added.+ "env": dict(env),})Path(cmd[cmd.index("papermill") + 2]).write_text("{}", encoding = "utf-8")@@ -115,7 +158,15 @@ def _drive(raised = Nonetry:for cell in driver["cells"][:2]:- source = "".join(cell["source"]).replace("/kaggle/working", str(tmp_path))+ source = (+ "".join(cell["source"])+ # Venvs moved off /kaggle/working (19.5 GB, and the artifact)+ # onto the ~1 TB overlay when two legs per card made four of+ # them possible at once. Both roots are rewritten here, or the+ # stub counts venvs in a directory nothing ever writes to.+ .replace("/tmp/t4ci_venvs", str(tmp_path / "venvs"))+ .replace("/kaggle/working", str(tmp_path))+ )try:exec(compile(source, "<driver-cell>", "exec"), namespace)except SystemExit as exc:@@ -173,6 +224,7 @@ class _PackedStub(_Stub):gpus,durations = None,hold = 0.05,+ vram = None,):super().__init__(gpus = gpus)self.durations = durations or {}@@ -180,35 +232,101 @@ class _PackedStub(_Stub):self._live_on_card: dict = {}self._lock = threading.Lock()self.same_card_overlaps: list = []+ self.peak_card_gb: dict = {}+ self.peak_card_legs: dict = {}+ self.vram = vram or {}self.max_live_venvs = 0self.root: Path | None = None+ self.venv_root: Path | None = None+ self.venvs_created: list = []def run(self, cmd, **kw):cmd = [str(c) for c in cmd]if len(cmd) > 2 and cmd[1] == "venv":+ # Recorded at CREATION. Looking for leftover venv_* after the run+ # cannot tell where they were built: the teardown removes them, so+ # a kernel building every venv on the 19.5 GB artifact volume ends+ # just as clean as one building them on the big overlay.+ self.venvs_created.append(Path(cmd[2]))Path(cmd[2]).mkdir(parents = True, exist_ok = True)if "papermill" in cmd:notebook = Path(cmd[cmd.index("papermill") + 1]).namecard = (kw.get("env") or {}).get("CUDA_VISIBLE_DEVICES")with self._lock:- if self._live_on_card.get(card):- self.same_card_overlaps.append((card, self._live_on_card[card], notebook))- self._live_on_card[card] = notebook- if self.root is not None:- live = len(list(self.root.glob("venv_*")))- self.max_live_venvs = max(self.max_live_venvs, live)+ live = self._live_on_card.setdefault(card, set())+ live.add(notebook)+ # Two legs on one card is now LEGAL when their measured VRAM+ # fits, so the overlap itself is no longer the finding. What is+ # recorded is the peak SUM, which is the thing that has to stay+ # under budget -- an overlap of two 0.7 GB legs is the feature,+ # and an overlap involving gptoss at 12.78 GB is the bug.+ self.peak_card_gb[card] = max(+ self.peak_card_gb.get(card, 0.0),+ sum(self.vram.get(n, 1.0) for n in live),+ )+ self.peak_card_legs[card] = max(self.peak_card_legs.get(card, 0), len(live))+ if len(live) > 1:+ self.same_card_overlaps.append((card, sorted(live)))+ if self.venv_root is not None:+ self.max_live_venvs = max(+ self.max_live_venvs, len(list(self.venv_root.glob("venv_*")))+ )time.sleep(self.durations.get(notebook, self.hold))with self._lock:- self._live_on_card[card] = None+ self._live_on_card[card].discard(notebook)return super().run(cmd, **kw)+class _HubStub(types.ModuleType):+ """Records `snapshot_download` calls in order, with a hold.++ The hold is not decoration. The prefetch runs on a thread nobody joins, so+ an instant stub would let it finish before the first card even starts and+ every ordering question this file asks would answer itself trivially.+ """++ def __init__(+ self,+ hold = 0.02,+ fail_for = (),+ ):+ super().__init__("huggingface_hub")+ self.calls: list = []+ self.hold = hold+ self.fail_for = set(fail_for)+ self.hf_home_at_call: list = []+ # What the hub was asked to FILTER on, per call. Recorded because the+ # patterns are computed a long way from here and reported in the+ # summary, and a version that worked out the right glob and then never+ # passed it would look identical in every artifact.+ self.patterns_at_call: list = []+ self._lock = threading.Lock()++ def snapshot_download(+ self,+ repo_id = None,+ **kw,+ ):+ with self._lock:+ self.calls.append(repo_id)+ self.hf_home_at_call.append(os.environ.get("HF_HOME"))+ self.patterns_at_call.append(kw.get("allow_patterns"))+ time.sleep(self.hold)+ if repo_id in self.fail_for:+ raise RuntimeError(f"stub refuses {repo_id}")++def _drive_packed(tmp_path,leg_names,*,gpus,durations = None,+ studio = None,+ prefetch_repos = (),+ hub = None,+ after_gpu_concurrent = False,+ venv_fallback = False,):driver = build_kernel.build_kernel(SMOKE_DIR,@@ -218,43 +336,184 @@ def _drive_packed(extra_args = (),per_run_timeout = 60,skip_reference = True,+ studio = studio,+ prefetch_repos = prefetch_repos,+ after_gpu_concurrent = after_gpu_concurrent,+ )+ stub = _PackedStub(+ gpus = gpus,+ durations = durations,+ vram = {f"t4_{n}.ipynb": LEGS[n].vram_gb for n in leg_names},)- stub = _PackedStub(gpus = gpus, durations = durations)stub.root = tmp_path+ # On the fallback path the venvs land in WORK itself, so that is where the+ # stub has to count them.+ stub.venv_root = tmp_path if venv_fallback else tmp_path / "venvs"+ hub = hub if hub is not None else _HubStub()saved = sys.modules["subprocess"]+ saved_hub = sys.modules.get("huggingface_hub")sys.modules["subprocess"] = stub+ sys.modules["huggingface_hub"] = hubnamespace: dict = {}raised = Nonetry:for cell in driver["cells"][:2]:- source = "".join(cell["source"]).replace("/kaggle/working", str(tmp_path))+ source = (+ "".join(cell["source"])+ # Venvs moved off /kaggle/working (19.5 GB, and the artifact)+ # onto the ~1 TB overlay when two legs per card made four of+ # them possible at once. Both roots are rewritten here, or the+ # stub counts venvs in a directory nothing ever writes to.+ # venv_fallback points the preferred root at a path whose+ # parent is a regular file, so `mkdir` raises OSError and the+ # kernel takes its own fallback branch. Rewriting it straight+ # to WORK would test an assignment; this tests the branch.+ .replace(+ "/tmp/t4ci_venvs",+ str(tmp_path / "blocked" / "t4ci_venvs")+ if venv_fallback+ else str(tmp_path / "venvs"),+ )+ .replace("/kaggle/working", str(tmp_path))+ )try:exec(compile(source, "<driver-cell>", "exec"), namespace)except SystemExit as exc:raised = excbreak+ # JOIN the lane before handing back. The kernel deliberately does not+ # (it is a daemon thread, so a slow download cannot hold the session+ # open), but a test that merely SAMPLES it leaks: the lane resolves+ # `huggingface_hub` out of sys.modules at call time, so one still+ # running after its test returns records into the NEXT test's stub.+ # That is not hypothetical -- it is how this harness first went order+ # dependent, passing alone and failing inside the suite.+ lane = namespace.get("prefetch_thread")+ if lane is not None:+ lane.join(30.0)+ assert not lane.is_alive(), "the prefetch lane outlived its test"finally:sys.modules["subprocess"] = saved- return {"stood_down": raised, "stub": stub, "results": namespace.get("results") or {}}+ if saved_hub is None:+ sys.modules.pop("huggingface_hub", None)+ else:+ sys.modules["huggingface_hub"] = saved_hub+ return {+ "stood_down": raised,+ "stub": stub,+ "hub": hub,+ "results": namespace.get("results") or {},+ "card_load": namespace.get("card_load") or {},+ "card_count": namespace.get("card_count") or {},+ }-ALL_FOUR = ["gptoss", "frontier", "canary", "control"]+# Derived from KERNELS, not a second copy of it. As a literal this silently+# went on describing the OLD longest-first order after legs.py moved to the+# second-wave one, so every test driving it was exercising an order the kernel+# no longer builds -- including the test that exists to assert the order.+# Derived from KERNELS, so it follows the registry rather than restating it.+# Renamed from ALL_LEGS when the Default leg made the kernel five: a name that+# counts is a name that goes stale silently, and two assertions below had+# already hardcoded the 4 to match it.+# Every wired leg, INCLUDING the all-card one, because the scheduling rules+# below have to hold for the set that actually ships.+#+# This was narrowed to exclude all-card legs for one commit, on the reasoning+# that a leg outside the card queue cannot answer a question about who shares a+# card. The reasoning was half right and the narrowing was hiding a bug: the+# all-card leg was charging a card_count slot, so it really did cost the queue+# a slot on a card, and the co-tenancy rule really was failing. The count is+# global now and the rule holds with the leg present, so the wider set is back.+ALL_LEGS = list(KERNELS[0])+++def test_losing_tmp_drops_the_kernel_back_to_one_leg_per_card(tmp_path):+ """The venv fallback described an intention nothing implemented.++ Venvs moved to /tmp because co-scheduling made four torch-bearing venvs+ possible at once and four do not fit in the 19.5 GB /kaggle/working. The+ fallback for a box with no writable /tmp says it "keeps a one-leg-per-card+ run working" -- but MAX_LEGS_PER_CARD was a constant, so the fallback put+ the venvs back on the small partition and went right on building two per+ card. It would have surfaced as an install dying halfway through, which+ reads like anything except a full disk.++ The fallback BRANCH is exercised, not simulated: the preferred root is+ pointed at a path whose parent is a regular file, so mkdir raises exactly+ as it would there.+ """+ (tmp_path / "blocked").write_text("not a directory")+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2, venv_fallback = True)+ assert driven["stood_down"] is None+ stub = driven["stub"]+ assert stub.venv_root is not None+ for card, count in stub.peak_card_legs.items():+ assert count <= 1, (+ f"card {card} ran {count} legs at once with the venvs back on "+ f"/kaggle/working: {stub.same_card_overlaps}"+ )+ assert stub.max_live_venvs <= 2, stub.max_live_venvs+ assert len(stub.papermill) == len(ALL_LEGS), stub.papermill-def test_four_legs_on_two_cards_never_put_two_legs_on_one_card_at_once(tmp_path):- """The property that makes packing safe at all.+def test_a_seeds_seat_is_taken_before_any_worker_can_look_at_the_card(tmp_path):+ """The race that put 13.48 GB on a 13.0 GB card, on real hardware.- Four payloads across two T4s is only sound because a card takes its next- leg when the previous one has EXITED. If they overlapped, each child would- still pass its own `device_count() == 1` assertion and then fight for 15GB,- which is exactly the failure the shortfall guard was added to prevent and- exactly the one it cannot see from where it stands.+ `test_no_card_is_ever_asked_to_hold_more_than_it_has` asserts the same+ budget and passed throughout, because with default stub durations every+ leg finishes before the 5s start stagger elapses and no overlap is ever+ recorded. Run 32667451396 was not so lucky: gpu1's seed sat unreserved for+ those 5s, a free worker saw an empty card and put gptoss on it, and when+ the seed worker finally woke, `_admit` correctly refused and the caller+ threw the answer away. control and gptoss then shared one card for 691s.++ So the durations here are chosen to hold the window open rather than to be+ fast: gptoss must outlive the stagger, or the second leg lands after it has+ already finished and the test goes green on a schedule that never happened.+ """+ driven = _drive_packed(+ tmp_path,+ ALL_LEGS,+ gpus = 2,+ durations = {+ "t4_canary.ipynb": 2.0,+ "t4_control.ipynb": 2.0,+ "t4_frontier.ipynb": 0.2,+ "t4_gptoss.ipynb": 7.0,+ },+ )+ stub = driven["stub"]+ for card, peak in stub.peak_card_gb.items():+ assert peak <= 13.0, f"card {card} peaked at {peak} GB. Overlaps: {stub.same_card_overlaps}"+ # gptoss is 12.78 of a 13.0 budget, so it can only ever run alone. Asserted+ # on the overlap record as well as the sum: a VRAM table that silently+ # under-priced it would satisfy the sum check while the card burned.+ for card, live in stub.same_card_overlaps:+ assert "t4_gptoss.ipynb" not in live, (card, live)+++def test_no_card_is_ever_asked_to_hold_more_than_it_has(tmp_path):+ """Was "never two legs on one card"; is now "never over the VRAM budget".++ Two legs on a card is the FEATURE, not the bug: measured on run+ 32611343797 the three Qwen legs peak at 0.70 GB each on a 14.56 GB card,+ so one leg per card left it 95% empty. What must still never happen is the+ thing that produced the OOM this file's shortfall guard was written for --+ payloads whose summed appetite exceeds the card. gptoss peaks at 12.78 GB,+ so it is excluded by the arithmetic rather than by a special case.++ Asserted on the summed GB and not on the overlap, because after this change+ an overlap is exactly what success looks like."""- driven = _drive_packed(tmp_path, ALL_FOUR, gpus = 2)+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2)assert driven["stood_down"] is Nonestub = driven["stub"]- assert stub.same_card_overlaps == [], stub.same_card_overlaps- assert len(stub.papermill) == 4, stub.papermill+ for card, peak in stub.peak_card_gb.items():+ assert peak <= 13.0, f"card {card} peaked at {peak} GB: {stub.same_card_overlaps}"+ for card, count in stub.peak_card_legs.items():+ assert count <= 2, f"card {card} held {count} legs at once"+ assert len(stub.papermill) == len(ALL_LEGS), stub.papermill# Both cards are used, and every leg ran. The SPLIT is deliberately not# asserted: how many legs each card ends up with is a function of how long# the legs take relative to the 5s venv stagger, not something the@@ -263,24 +522,59 @@ def test_four_legs_on_two_cards_never_put_two_legs_on_one_card_at_once(tmp_path)# and the split is 2/2; under the sub-second stubs here the first card# legitimately drains most of the queue before the second clears its# stagger. Pinning 2/2 would be pinning the stub's timing.- assert set(p["cuda"] for p in stub.papermill) == {"0", "1"}, stub.papermill---def test_the_longest_leg_starts_first_so_the_schedule_can_balance_around_it(tmp_path):- """Start order is longest-first, and it is load bearing rather than tidy.-- Measured on run 32607621452: gptoss 384.1s, frontier 312.2s, canary 265.3s,- control 262.2s. Longest-first packs those as 646.3s, which is the optimal- split of the four; `sorted(PAYLOADS)` would start gptoss LAST, and a greedy- scheduler cannot balance around the leg that sets the makespan if it picks- it up at the end.+ # Both cards pinned, across the payloads that HAVE a card index. An+ # all-card leg is unpinned by design -- that is the whole reason it exists,+ # since unsloth binds its DEVICE_COUNT > 1 helpers on visibility -- so it+ # inherits the ambient CUDA_VISIBLE_DEVICES exactly as the Studio lanes do,+ # and asserting it against a card index tests the box this ran on.+ pinned = [+ p+ for p in stub.papermill+ if p["notebook"] not in {f"t4_{LEGS[n].name}.ipynb" for n in ALL_LEGS if LEGS[n].all_cards}+ ]+ assert set(p["cuda"] for p in pinned) == {"0", "1"}, stub.papermill+ # ...and if any all-card leg is WIRED, it really did run and really did run+ # unpinned. Conditional because the wired set has none today: multi_gpu was+ # measured and held out. Asserting one unconditionally would go red on a+ # scheduling change that is nothing to do with this rule.+ unpinned = [p for p in stub.papermill if p not in pinned]+ if any(LEGS[n].all_cards for n in ALL_LEGS):+ assert unpinned, "an all-card leg is wired and no unpinned payload ran"+ else:+ assert not unpinned, unpinned+++def test_gptoss_starts_in_the_second_wave_so_the_prefetch_has_a_window(tmp_path):+ """Was longest-first; is now second-wave, and the change is the point.++ A prefetch only pays for what it finishes BEFORE the leg that wants the+ model starts, and gptoss is the only leg with a ~12 GB download. Starting+ it at t=0 leaves no window in front of it, which is why prefetching without+ this reorder measures WORSE than doing neither (603.1s against 563.1s).++ Third is first pick of the second wave: with two cards, positions 0 and 1+ are seeded and 2 is the first to be taken off the pending queue, so gptoss+ starts at ~190-220s under the measured durations. That is lead time the+ prefetch spends, and a small leg is still running beside it.++ Asserted on POSITION in the order rather than on a start timestamp: the+ timestamp is a function of the stub's durations, and pinning it would pin+ the stub. legs.KERNELS carries the full table."""- driven = _drive_packed(tmp_path, ALL_FOUR, gpus = 2)+ order = list(KERNELS[0])+ assert order.index("gptoss") == 2, (+ f"gptoss is at position {order.index('gptoss')} of {order}; first means "+ "the prefetch has no window and second-wave is what buys the saving"+ )+ # ...and it must not be LAST either, which is the other intuitive answer.+ # gptoss is the longest leg, so ending on it idles the other card for its+ # whole ~284s: simulated at 651.1s worst case against 528.1s here.+ assert order[-1] != "gptoss", order++ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2)started = [p["notebook"] for p in driven["stub"].papermill]- assert started[0] == "t4_gptoss.ipynb", started- assert started != sorted(- started- ), "payloads are running in alphabetical order, so the longest leg is last"+ assert started[0] != "t4_gptoss.ipynb", started+ assert started != sorted(started), "payloads are running in alphabetical order"def test_each_leg_keeps_its_own_venv_compile_cache_and_ipykernel(tmp_path):@@ -292,7 +586,7 @@ def test_each_leg_keeps_its_own_venv_compile_cache_and_ipykernel(tmp_path):an index reused across a wave would silently merge two legs' trees and thelast writer would win."""- driven = _drive_packed(tmp_path, ALL_FOUR, gpus = 2)+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2)calls = driven["stub"].papermillfor field in ("kernel", "compile_location", "notebook"):values = [c[field] for c in calls]@@ -300,19 +594,220 @@ def test_each_leg_keeps_its_own_venv_compile_cache_and_ipykernel(tmp_path):def test_a_finished_leg_gives_its_virtualenv_back(tmp_path):- """Otherwise four torch trees sit on /kaggle/working at once.+ """Each venv carries its own torch and its own NVIDIA runtime.The tail cell prunes `venv_*`, but only after every payload has finished,- which was sufficient when a kernel held one payload per card. Packed, the- peak is what matters, and it has to stay at one venv per CARD rather than- one per LEG.+ so the PEAK is what matters and freeing at the end of each leg is what+ bounds it. The bound is one venv per concurrent LEG, which co-scheduling+ raised from 2 to 4.++ That raise is precisely why the venvs no longer live on `/kaggle/working`.+ That path is 19.5 GB and is also what Kaggle ships home; four torch trees+ do not fit in it, and the failure would arrive as an install dying midway+ for reasons that look nothing like a full disk. They go on the ~1 TB+ overlay instead, and only the evidence stays where Kaggle collects it."""- driven = _drive_packed(tmp_path, ALL_FOUR, gpus = 2)+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2)stub = driven["stub"]+ ceiling = 2 * 2 # cards x MAX_LEGS_PER_CARDassert (- stub.max_live_venvs <= 2- ), f"{stub.max_live_venvs} virtualenvs were alive at once on a 2-card kernel"- assert list(tmp_path.glob("venv_*")) == [], "a payload left its virtualenv behind"+ stub.max_live_venvs <= ceiling+ ), f"{stub.max_live_venvs} virtualenvs were alive at once, ceiling {ceiling}"+ assert list((tmp_path / "venvs").glob("venv_*")) == [], "a payload left its virtualenv behind"+ # ...and none of them was ever created on the artifact volume.+ assert stub.venvs_created, "no virtualenv was built at all"+ for created in stub.venvs_created:+ assert (tmp_path / "venvs") in created.parents, (+ f"a virtualenv was built at {created}, on /kaggle/working -- which "+ "is 19.5 GB and is also the artifact Kaggle ships home"+ )+++# --------------------------------------------------- Studio in the same kernel++STUDIO = {+ "unsloth_ref": "main",+ "repo_url": "https://github.com/unslothai/unsloth",+ "payload_args": "--max-steps 8",+}+STUDIO_INSTALL = build_kernel.STUDIO_INSTALL_NOTEBOOK+STUDIO_TEST = build_kernel.STUDIO_TEST_NOTEBOOK+# A value no card index can be confused with, so "the driver left this alone"+# and "the driver pinned a card" are distinguishable. An unpinned lane inherits+# whatever the ambient environment has; a pinned one is overwritten with a+# single index.+AMBIENT_CUDA = "0,1"+++def _drive_with_studio(+ tmp_path,+ monkeypatch,+ leg_names,+ *,+ gpus = 2,+ durations = None,+):+ monkeypatch.setenv("CUDA_VISIBLE_DEVICES", AMBIENT_CUDA)+ return _drive_packed(tmp_path, leg_names, gpus = gpus, durations = durations, studio = STUDIO)+++def test_the_studio_install_never_takes_a_card_and_the_legs_never_wait_for_it(+ tmp_path, monkeypatch+):+ """The whole point of carrying Studio here: its install is free time.++ Checkout, `install.sh --local`, the frontend build and the Playwright+ browser are network and CPU and touch no GPU, so they run beside the+ training legs rather than after them. If this lane ever queued for a card+ it would displace a leg and the merge would cost more than it saves.+ """+ driven = _drive_with_studio(+ tmp_path,+ monkeypatch,+ ALL_LEGS,+ durations = {f"t4_{LEGS[n].name}.ipynb": 0.30 for n in ALL_LEGS},+ )+ assert driven["stood_down"] is None+ calls = {c["notebook"]: c for c in driven["stub"].papermill}+ assert STUDIO_INSTALL in calls, sorted(calls)++ # Unpinned. install.sh --local resolves torch, and an installer that cannot+ # see a device resolves a CPU-only one -- which is the exact regression+ # Studio's verify cell exists to catch, so hiding the cards here would+ # manufacture it.+ assert calls[STUDIO_INSTALL]["cuda"] == AMBIENT_CUDA, calls[STUDIO_INSTALL]+ # ...while every leg is still pinned to exactly one card. The SPLIT is not+ # asserted, for the reason given in+ # test_four_legs_on_two_cards_never_put_two_legs_on_one_card_at_once: how+ # many legs each card takes depends on leg duration against the 5s venv+ # stagger, and under sub-second stubs the first card legitimately drains+ # most of the queue.+ # "every leg" here means every leg that HAS a card. An all-card leg is+ # unpinned by design and inherits the ambient CUDA_VISIBLE_DEVICES, like the+ # Studio lanes above; asserting it against a card index tests the box the+ # suite ran on rather than the scheduler.+ unpinned = {f"t4_{LEGS[n].name}.ipynb" for n in ALL_LEGS if LEGS[n].all_cards}+ leg_cards = [c["cuda"] for n, c in calls.items() if n.startswith("t4_") and n not in unpinned]+ assert len(leg_cards) == len(ALL_LEGS) - len(unpinned), calls+ assert set(leg_cards) == {"0", "1"}, leg_cards+ # Every all-card leg really did run, and really did run unpinned. Without+ # this the exclusion above could quietly cover a leg that never started.+ for notebook in unpinned:+ assert notebook in calls, sorted(calls)+ assert calls[notebook]["cuda"] == AMBIENT_CUDA, calls[notebook]+ # Two legs on a card is legal now (see the VRAM budget); what must hold is+ # that the summed appetite never exceeds what the card has.+ for card, peak in driven["stub"].peak_card_gb.items():+ assert peak <= 13.0, (card, peak, driven["stub"].same_card_overlaps)+++def test_the_studio_assertions_wait_for_both_cards_rather_than_borrowing_one(tmp_path, monkeypatch):+ """Studio keeps both T4s visible, and that is deliberate upstream.++ Its own driver says so: "Studio's own device selection is part of what is+ under test; masking one would test a machine nobody has." So the GPU half+ runs once the leg queue has drained, unpinned, rather than being handed a+ single card out of the queue.+ """+ driven = _drive_with_studio(tmp_path, monkeypatch, ALL_LEGS)+ calls = [c["notebook"] for c in driven["stub"].papermill]+ assert STUDIO_TEST in calls, calls+ # Last, after every leg.+ assert calls[-1] == STUDIO_TEST, calls+ by_name = {c["notebook"]: c for c in driven["stub"].papermill}+ assert by_name[STUDIO_TEST]["cuda"] == AMBIENT_CUDA, by_name[STUDIO_TEST]+++def test_a_failed_studio_install_skips_its_assertions_with_the_reason(tmp_path, monkeypatch):+ """Otherwise the missing venv is reported as a Studio regression.++ The install half is what puts the interpreter, the frontend and the+ llama.cpp on disk. Running the assertions against a half-built tree fails+ on `no interpreter at ...`, which reads like the code under test broke+ rather than like the install did.+ """+ monkeypatch.setenv("CUDA_VISIBLE_DEVICES", AMBIENT_CUDA)++ class _InstallFails(_PackedStub):+ def run(self, cmd, **kw):+ cmd = [str(c) for c in cmd]+ if "papermill" in cmd and STUDIO_INSTALL in " ".join(cmd):+ self.papermill.append(+ {+ "notebook": STUDIO_INSTALL,+ "cuda": None,+ "kernel": None,+ "compile_location": None,+ }+ )+ Path(cmd[cmd.index("papermill") + 2]).write_text("{}", encoding = "utf-8")+ return types.SimpleNamespace(returncode = 1, stdout = "", stderr = "")+ return super().run(cmd, **kw)++ driver = build_kernel.build_kernel(+ SMOKE_DIR,+ ALL_LEGS,+ unsloth_ref = "main",+ zoo_ref = "main",+ extra_args = (),+ per_run_timeout = 60,+ skip_reference = True,+ studio = STUDIO,+ )+ stub = _InstallFails(gpus = 2)+ stub.root = tmp_path+ stub.venv_root = tmp_path / "venvs"+ saved = sys.modules["subprocess"]+ sys.modules["subprocess"] = stub+ namespace: dict = {}+ try:+ for cell in driver["cells"][:2]:+ source = (+ "".join(cell["source"])+ # Venvs moved off /kaggle/working (19.5 GB, and the artifact)+ # onto the ~1 TB overlay when two legs per card made four of+ # them possible at once. Both roots are rewritten here, or the+ # stub counts venvs in a directory nothing ever writes to.+ .replace("/tmp/t4ci_venvs", str(tmp_path / "venvs"))+ .replace("/kaggle/working", str(tmp_path))+ )+ exec(compile(source, "<driver-cell>", "exec"), namespace)+ finally:+ sys.modules["subprocess"] = saved++ ran = [c["notebook"] for c in stub.papermill]+ assert STUDIO_TEST not in ran, ran+ # Every leg still ran: a broken Studio install must not take the notebook+ # signal down with it.+ assert sorted(n for n in ran if n.startswith("t4_")) == sorted(+ f"t4_{LEGS[leg].name}.ipynb" for leg in ALL_LEGS+ )+ recorded = (namespace.get("results") or {}).get(STUDIO_TEST)+ assert recorded is not None, "the skip was not recorded at all"+ assert recorded["returncode"] is None+ assert "install lane did not succeed" in recorded["error"]+++def test_studio_is_not_in_the_card_queue(tmp_path, monkeypatch):+ """ORDER is the legs. Either Studio half in it would be handed a card."""+ driver = build_kernel.build_kernel(+ SMOKE_DIR,+ ALL_LEGS,+ unsloth_ref = "main",+ zoo_ref = "main",+ extra_args = (),+ per_run_timeout = 60,+ skip_reference = True,+ studio = STUDIO,+ )+ setup = "".join(driver["cells"][0]["source"])+ order = next(l for l in setup.splitlines() if l.startswith("ORDER = "))+ assert STUDIO_INSTALL not in order, order+ assert STUDIO_TEST not in order, order+ assert order.count("t4_") == len(ALL_LEGS), order+ # ...but both are carried, or the kernel would have nothing to run.+ payloads = set(driver["metadata"]["kaggle_t4_ci"]["payloads"])+ assert {STUDIO_INSTALL, STUDIO_TEST} <= payloads, sorted(payloads)def test_a_one_card_allocation_still_stands_a_packed_kernel_down(tmp_path):@@ -325,7 +820,7 @@ def test_a_one_card_allocation_still_stands_a_packed_kernel_down(tmp_path):infrastructure, because one card silently serialises the whole kernel anddoubles its wall clock while looking like a slow but healthy run."""- driven = _drive_packed(tmp_path, ALL_FOUR, gpus = 1)+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 1)assert driven["stood_down"] is not None, "a 1-GPU allocation ran the packed kernel anyway"assert driven["stub"].papermill == []@@ -1853,3 +2348,961 @@ def test_main_bounds_the_whole_evidence_phase_it_is_budgeted_for(monkeypatch, tmassert seen[0] is not None, "main() never handed the collection a deadline"assert seen[0] == seen[1], "the two kernels must share ONE budget, not get one each"assert seen[0] - started <= launch.EVIDENCE_BUDGET_SEC+++# --------------------------------------------------------- the merged kernel's+# --------------------------------------------------------- two reporters++NOTEBOOK_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "kaggle-t4-notebook-ci.yml"+++def test_the_merged_kernel_runs_both_reporters():+ """One kernel, two experiments, so two report steps over one evidence dir.++ Dropping the Studio one is the failure this guard exists for: the kernel+ would still install Studio, still drive the UI on a T4, and the job would+ still go green with nothing said about it. The T4 reporter FILTERS the+ studio-gpu label out, so its section would look complete while the payload+ that half the wall clock went on is unreported.+ """+ source = NOTEBOOK_WORKFLOW.read_text(encoding = "utf-8")+ assert ".github/scripts/kaggle_t4_ci/report.py" in source+ assert ".github/scripts/kaggle_studio_ci/report.py" in source+ assert ".github/scripts/kaggle_studio_ci/collect_evidence.py" in source+++def test_the_shared_wheels_are_the_specs_every_leg_holds_in_common():+ """Built once from the very SHAs the legs name -- main AND the ref tested.++ Every leg installs unsloth_zoo and unsloth from the same two pinned SHAs,+ and pip does not cache a VCS build, so run 32679427416 cloned and built+ both FOUR times: install was 149-191s per leg, the largest single phase of+ each and 41% of gpt-oss.++ The list is DERIVED from the legs' own groups, never declared again, and+ that is what keeps it honest. A hand-written copy of the two SHAs would+ drift silently, because a wheel built from the wrong ref installs perfectly+ and fails nothing at all.++ The intersection is the safety property, not an optimisation: a spec only+ one leg carries is part of what that leg tests, and sharing it would make+ the legs agree about the thing they exist to disagree about.+ """+ common = build_kernel._shared_vcs_specs(+ {+ "a": [["unsloth_zoo @ git+u@S1"], ["transformers==5.5.0"]],+ "b": [["unsloth_zoo @ git+u@S1"], ["--upgrade", "transformers"]],+ }+ )+ assert common == ("unsloth_zoo @ git+u@S1",), common++ # Different refs for the same package share NOTHING. Returning either one+ # would hand a leg a wheel built from the other leg's commit.+ assert (+ build_kernel._shared_vcs_specs(+ {+ "a": [["unsloth @ git+u@S1"]],+ "b": [["unsloth @ git+u@S2"]],+ }+ )+ == ()+ )++ # A spec only one leg carries is never shared.+ assert build_kernel._shared_vcs_specs(+ {+ "a": [["x @ git+u@S1"], ["y @ git+u@S9"]],+ "b": [["x @ git+u@S1"]],+ }+ ) == ("x @ git+u@S1",)++ # And on the real legs: BOTH packages, at the refs asked for.+ driver = build_kernel.build_kernel(+ SMOKE_DIR,+ ALL_LEGS,+ unsloth_ref = "PRSHA",+ zoo_ref = "MAINSHA",+ extra_args = (),+ per_run_timeout = 60,+ skip_reference = True,+ shared_wheels = True,+ )+ src = "".join("".join(c["source"]) for c in driver["cells"])+ specs = re.search(r"SHARED_WHEEL_SPECS = (.+)", src).group(1)+ assert "unsloth @ git+" in specs and "unsloth_zoo @ git+" in specs, specs+ assert "PRSHA" in specs and "MAINSHA" in specs, specs+ # Built before any leg can start. A background build loses the race it+ # exists to win: the legs are admitted and start installing at t=0.+ assert src.index('"pip", "wheel"') < src.index(+ "threads = []"+ ), "the wheels are built after the leg workers start, so no leg can use them"+++def test_every_leg_gets_its_own_torch_and_triton_cache(tmp_path):+ """A separate venv never protected these, and nobody noticed.++ UNSLOTH_COMPILE_LOCATION was set per leg and assumed to be the whole story.+ torch and triton key their caches off $TMPDIR rather than off the+ interpreter: on torch 2.9.1 an unset TORCHINDUCTOR_CACHE_DIR resolves to+ `tempfile.gettempdir()/torchinductor_$USER`+ (torch/_inductor/runtime/cache_dir_utils.py:22) and the triton cache lands+ under that same directory. So four legs whose entire purpose is to install+ DIFFERENT transformers/TRL/peft versions and compile the same modules were+ sharing one /tmp/torchinductor_root.++ Asserted as DISTINCT per leg rather than merely present -- one directory+ named once and handed to everybody would satisfy "is set" and reproduce the+ bug exactly.+ """+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2)+ seen: dict[str, set] = {}+ for call in driven["stub"].papermill:+ env = call.get("env") or {}+ for key in (+ "TORCHINDUCTOR_CACHE_DIR",+ "TRITON_CACHE_DIR",+ "TMPDIR",+ "UNSLOTH_COMPILE_LOCATION",+ ):+ assert env.get(key), f"{call['notebook']} has no {key}"+ seen.setdefault(key, set()).add(env[key])+ for key, values in seen.items():+ assert len(values) == len(+ driven["stub"].papermill+ ), f"{key} is shared between legs: {sorted(values)}"+++def test_two_dispatches_can_hold_the_two_kaggle_slots_at_once():+ """One concurrency group cannot express "at most two", and one was used.++ A Kaggle account allows 2 concurrent GPU sessions and this job takes one,+ so the cap is 2 -- but GitHub concurrency is 1 per group. The single group+ did not merely serialise: only ONE run may be PENDING in a group, so a+ second queued run CANCELS the first instead of queueing behind it. That+ killed run 32674255736 and made an A/B impossible to run at all, which is+ how two "different" configurations came to be compared against each other+ while executing the same schedule.++ Bounded by construction is the property worth guarding: the input offers+ exactly two slots, so the account can never be asked for a third session.+ """+ source = NOTEBOOK_WORKFLOW.read_text(encoding = "utf-8")+ workflow = yaml.safe_load(source)+ slot = workflow[True]["workflow_dispatch"]["inputs"]["slot"]+ assert slot["type"] == "choice", slot+ assert slot["options"] == ["1", "2"], (+ f"the slot input offers {slot['options']}, so the account could be "+ f"asked for more than its 2 concurrent sessions"+ )+ assert slot.get("default") == "1", slot++ # BOTH levels, because the discard rule applies to both and fixing only the+ # job left the whole thing broken in exactly the same way: the two runs+ # still shared one workflow-level group, so the second dispatch discarded+ # the first while it was pending and never reached the job group at all.+ for scope, block in (+ ("workflow", workflow["concurrency"]),+ ("job", workflow["jobs"]["t4-smoke"]["concurrency"]),+ ):+ group = block["group"]+ assert "inputs.slot" in group, f"{scope}: {group}"+ # A non-dispatch event has no input and must land in a single shared+ # slot, or every push would get a session of its own.+ assert "'1'" in group, f"{scope}: {group}"+ assert block["cancel-in-progress"] is False, scope+++def test_the_shared_wheel_build_is_opt_in():+ """Measured once, attributable to nothing, so it ships behind a flag.++ On run 32689629906 the wheels helped the leg that runs ALONE (gpt-oss+ install 191.2s -> 152.6s) and cost the three that run CONCURRENTLY+ (149-163s -> 319-334s). That run also changed the torch/triton cache+ layout, so neither effect can be attributed to either change. A default-on+ optimisation resting on that would be a guess wearing a measurement's+ clothes.+ """+ source = NOTEBOOK_WORKFLOW.read_text(encoding = "utf-8")+ workflow = yaml.safe_load(source)+ inputs = workflow[True]["workflow_dispatch"]["inputs"]+ assert inputs["shared_wheels"].get("default") is False, inputs["shared_wheels"]+ build = source.split("build_kernel.py")[1].split("- name:")[0]+ assert "$SHARED_WHEELS" in build, build+ assert "'--shared-wheels'" in source++ # Off means no wheel build in the kernel at all, not merely an unused one.+ off = build_kernel.build_kernel(+ SMOKE_DIR,+ ALL_LEGS,+ unsloth_ref = "R",+ zoo_ref = "R",+ extra_args = (),+ per_run_timeout = 60,+ skip_reference = True,+ )+ src = "".join("".join(c["source"]) for c in off["cells"])+ assert "SHARED_WHEEL_SPECS = ()" in src, (+ "wheels are off but the kernel still carries specs, so it would spend "+ "the build time and change nothing"+ )+++def test_the_workflow_can_actually_reach_studio_concurrent():+ """A CLI flag nothing passes is dead code that reads as a feature.++ This is not hypothetical. Run 32674263571 was dispatched as the VARIANT of+ an A/B on exactly this behaviour. `--studio-concurrent` existed in+ build_kernel's argument parser and was threaded all the way to+ AFTER_GPU_CONCURRENT, the unit tests for it passed, and the workflow never+ passed the flag -- so the kernel built with it False, the "variant" ran the+ control's schedule, and the comparison was a configuration against itself.+ Nothing was red. `AFTER_GPU_SHARED` was simply absent from kernel.log, and+ absence is not something a green tick reports.++ So the chain is asserted end to end: the input exists, something converts+ it into the flag, and the flag reaches the build command.+ """+ source = NOTEBOOK_WORKFLOW.read_text(encoding = "utf-8")+ workflow = yaml.safe_load(source)+ inputs = workflow[True]["workflow_dispatch"]["inputs"]+ assert "studio_concurrent" in inputs, sorted(inputs)+ # ON by default now, and the reason is a measurement: Studio's GPU half is+ # ~1100s and it used to start only once every leg had freed both cards, on+ # top of a ~1200s leg phase, so sharing takes that block off the critical+ # path. What it costs -- Studio seeing one T4 rather than two -- is not+ # silent: the payload sizes tensor_split to the visible cards and records+ # `tensor_split_over_two_cards: false` with a note, which is asserted by+ # test_studio_server_flags.py.+ assert inputs["studio_concurrent"].get("default") is True, inputs["studio_concurrent"]+ # And it must still be switchable OFF, or the both-cards run is unreachable.+ assert 'inputs.studio_concurrent }}" = "false"' in source, (+ "nothing reads the input as a way to turn sharing off, so a dispatch "+ "asking for the two-card coverage would silently get the shared shape"+ )++ build = source.split("build_kernel.py")[1].split("- name:")[0]+ assert "$STUDIO_CONCURRENT" in build, (+ "the build command does not interpolate STUDIO_CONCURRENT, so the "+ "input cannot reach the kernel no matter what it is set to"+ )+ assert "inputs.studio_concurrent" in source+ assert "'--studio-concurrent'" in source or '"--studio-concurrent"' in source++ # And the flag the workflow spells must be one the CLI accepts. A rename on+ # either side would otherwise land as an unrecognised argument at build+ # time, or worse, be silently ignored.+ cli = (CI_DIR / "build_kernel.py").read_text(encoding = "utf-8")+ assert '"--studio-concurrent"' in cli, "build_kernel.py does not define the flag"+++def test_the_t4_reporter_is_told_the_leg_count_not_the_payload_count():+ """`payloads` counts Studio; `legs` does not, and this reporter drops it.++ Handing it `payloads` makes a complete four-leg result read as short by one+ forever: it filters the studio-gpu report out and then compares what is+ left against a number that included it.+ """+ source = NOTEBOOK_WORKFLOW.read_text(encoding = "utf-8")+ reporter = source.split(".github/scripts/kaggle_t4_ci/report.py")[1].split("- name:")[0]+ assert "steps.build.outputs.legs" in reporter+ assert "steps.build.outputs.payloads" not in reporter+++ ("label", "reporter", "expect_red"),+ [+ ("control", "kaggle_t4_ci", True),+ ("control", "kaggle_studio_ci", False),+ ("studio-gpu", "kaggle_t4_ci", False),+ ("studio-gpu", "kaggle_studio_ci", True),+ ],+)+def test_a_failing_payload_only_reddens_the_reporter_that_owns_it(+ tmp_path, label, reporter, expect_red+):+ """The launcher writes ONE verdict for a kernel that now holds two+ unrelated experiments. A reporter reading it directly would announce a+ failure it cannot describe, over a section listing none, and point at the+ wrong half of a 13-minute kernel."""+ evidence = tmp_path / "evidence"+ evidence.mkdir()+ reports = [+ {"label": "control", "passed": label != "control", "steps": []},+ {"label": "studio-gpu", "passed": label != "studio-gpu", "assertions": []},+ ]+ (evidence / "launch_result.json").write_text(+ json.dumps(+ {+ "verdict": "fail",+ "reason": "1 of 2 payload(s) failed their assertions",+ "slug": "u/s",+ "kernel_state": "COMPLETE",+ "reports": reports,+ }+ )+ )+ proc = subprocess.run(+ [+ sys.executable,+ str(REPO_ROOT / ".github" / "scripts" / reporter / "report.py"),+ "--evidence",+ str(evidence),+ "--expect",+ "1",+ ],+ capture_output = True,+ text = True,+ )+ assert (proc.returncode == 1) is expect_red, proc.stdout+++def test_the_build_step_actually_packs_studio_in():+ """The whole pipelining claim in this workflow's header rests on one flag.++ Without it the kernel builds four legs, every reporter still renders, the+ Studio section reads NOT RUN with a plausible-sounding reason, and the job+ is green -- which is indistinguishable from a run whose sampling declined.+ """+ source = NOTEBOOK_WORKFLOW.read_text(encoding = "utf-8")+ build = source.split("- name: Build the kernel notebooks")[1].split("- name:")[0]+ assert "--with-studio" in build+ assert "--studio-args" in build+++# ------------------------------------------------------------- the prefetch lane+++def test_the_prefetch_lane_never_takes_a_card(tmp_path):+ """It is CPU and network work, and a card it held would be a card idle.++ The whole saving is that downloading happens BESIDE training rather than+ in front of it. A prefetch that consumed a GPU slot would move the wait+ rather than remove it, and would also break the packing arithmetic that+ assumes exactly two lanes compete for two cards.+ """+ hub = _HubStub()+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2, prefetch_repos = ("a/big", "b/small"), hub = hub)+ assert driven["stood_down"] is None+ assert hub.calls == ["a/big", "b/small"], hub.calls+ # Every papermill call is a LEG. The prefetch is not one of them, so it+ # cannot have been handed CUDA_VISIBLE_DEVICES.+ assert len(driven["stub"].papermill) == len(ALL_LEGS), driven["stub"].papermill+ # Two legs on a card is legal now (see the VRAM budget); what must hold is+ # that the summed appetite never exceeds what the card has.+ for card, peak in driven["stub"].peak_card_gb.items():+ assert peak <= 13.0, (card, peak, driven["stub"].same_card_overlaps)+++def test_the_leg_prefetch_does_not_redirect_hf_home(tmp_path):+ """The legs read the Kaggle image's DEFAULT cache.++ Pointing the lane at a private root is the silent failure this guards: it+ downloads all 12 GB perfectly, into a directory no leg looks in, reports+ success, and the run is green and no faster. Nothing at runtime would say+ so, which is why it is asserted here.+ """+ hub = _HubStub()+ before = os.environ.get("HF_HOME")+ _drive_packed(tmp_path, ALL_LEGS, gpus = 2, prefetch_repos = ("a/big",), hub = hub)+ assert hub.hf_home_at_call == [before], hub.hf_home_at_call+ assert os.environ.get("HF_HOME") == before+++def test_a_failing_prefetch_does_not_fail_the_kernel(tmp_path):+ """Graceful degradation is the entire safety argument for shipping this.++ The leg that wants the model still downloads it itself, exactly as it did+ before the lane existed, so a prefetch failure costs seconds. If it could+ fail the kernel it would be a brand new way to go red for something that+ is not under test -- on a payload that is not even the subject of the run.+ """+ hub = _HubStub(fail_for = ("a/big", "b/small"))+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2, prefetch_repos = ("a/big", "b/small"), hub = hub)+ assert driven["stood_down"] is None+ assert len(driven["stub"].papermill) == len(ALL_LEGS), driven["stub"].papermill+ assert all(r.get("returncode") == 0 for r in driven["results"].values()), driven["results"]+++def test_no_prefetch_repos_leaves_the_schedule_exactly_as_it_was(tmp_path):+ """The lane is opt-in at the call site, and off means OFF: no thread, no+ huggingface_hub import, no behaviour change for a kernel built without it."""+ hub = _HubStub()+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2, prefetch_repos = (), hub = hub)+ assert hub.calls == [], hub.calls+ assert driven["stood_down"] is None+ assert len(driven["stub"].papermill) == len(ALL_LEGS)+++def test_the_prefetch_list_matches_the_models_the_legs_actually_load():+ """A prefetch of the wrong repo downloads happily and warms nothing.++ There is no runtime feedback for this: the lane reports success, the legs+ download for themselves, and the only symptom is a saving that never+ arrives. So the declared list is checked against the DEFAULT_MODEL the+ payload scripts really carry, read out of their source.++ The DEFAULT_MODEL is where an earlier version of this test stopped, and+ stopping there is what let the bug through. What a leg ASKS FOR and what it+ LOADS are different for gpt-oss: `unsloth/gpt-oss-20b` is MXFP4, sm_75+ cannot read MXFP4, and unsloth redirects to `-unsloth-bnb-4bit` at load+ time. The old assertion compared the prefetch list against the declared+ name, so it agreed with a prefetch of 55.1 GB that no leg ever opened. It+ now applies LOAD_REDIRECTS first, and separately pins that the redirect it+ is applying is the one the payload actually documents.+ """+ from legs import LEGS, LOAD_REDIRECTS, PREFETCH_REPOS++ # Reading DEFAULT_MODEL out of two named scripts is where this stopped, and+ # it was too narrow twice over: it knew nothing of legs that pass --model on+ # the command line, and nothing of payloads beyond the two it names. The set+ # equality it then asserted was therefore satisfied by a prefetch list+ # missing the leg that SETS the makespan. The walk lives in+ # test_prefetch_covers_the_wired_legs.py, which owns the rule; this file+ # keeps the redirect-provenance half below, which that one does not cover.+ from test_prefetch_covers_the_wired_legs import models_for++ loaded = set()+ for leg in LEGS.values():+ loaded |= models_for(leg)+ stray = sorted(set(PREFETCH_REPOS) - loaded)+ assert not stray, f"prefetching {stray}, which no leg loads even after LOAD_REDIRECTS"++ # LOAD_REDIRECTS is only as good as its agreement with reality. If a+ # redirect stops being real, this list must stop claiming it -- or the+ # prefetch goes back to warming a cache nobody reads, in the other+ # direction and just as invisibly.+ #+ # There are two kinds of evidence for a redirect and they need different+ # checks. gpt-oss's target is written in the payload, so it can be read out+ # of the source. The Qwen one cannot: unsloth derives it at load time+ # through FLOAT_TO_INT_MAPPER and no file in this repo names it. Its only+ # evidence is a resolved_checkpoint field in a kernel report. So a redirect+ # no payload mentions must CITE the kernel that measured it, which is+ # checkable, rather than being taken on trust.+ sources = "".join(path.read_text(encoding = "utf-8") for path in sorted(SMOKE_DIR.glob("*.py")))+ legs_src = (Path(build_kernel.__file__).parent / "legs.py").read_text(encoding = "utf-8")+ for declared, actual in LOAD_REDIRECTS.items():+ if actual in sources:+ continue+ assert re.search(rf"#[^\n]*{re.escape(actual)}", legs_src) or any(+ actual in line and line.lstrip().startswith("#") for line in legs_src.splitlines()+ ), (+ f"LOAD_REDIRECTS says {declared} loads as {actual}, no payload "+ f"mentions {actual}, and no comment in legs.py cites the run that "+ f"measured it -- so the redirect is asserted and not observed"+ )+ # Qwen FIRST, and the reasoning inverted once the schedule was simulated+ # end to end. gpt-oss is bigger, but it is wanted by ONE leg whose setup+ # does not finish until ~160s anyway, whereas the small model gates THREE+ # legs and costs ~20s. Fetching the big one first pushes the small one out+ # past the moment the first leg is ready and delays three legs to give one+ # a head start it did not need.+ assert "Qwen" in PREFETCH_REPOS[0], PREFETCH_REPOS+ assert "gpt-oss" in PREFETCH_REPOS[-1], PREFETCH_REPOS+++def test_the_generated_prefetch_cell_runs_not_merely_compiles():+ """Compiling it is not enough, and that is not hypothetical here.++ The first version interpolated `hf_home` with `json.dumps`, so `None`+ became the JSON literal `null`. It compiled cleanly and died with a+ NameError the first time it RAN -- which on the real thing means minutes+ into a paid Kaggle session.+ """+ prefetch = build_kernel._prefetch_builder()+ hub = _HubStub(hold = 0.0)+ saved = sys.modules.get("huggingface_hub")+ sys.modules["huggingface_hub"] = hub+ try:+ for hf_home in (None, "/tmp/somewhere"):+ source = prefetch.prefetch_cell(+ ["a/b"], hf_home = hf_home, attempt_timeout = 2, total_timeout = 5+ )+ exec(compile(source, "<prefetch>", "exec"), {"__name__": "prefetch"})+ finally:+ if saved is None:+ sys.modules.pop("huggingface_hub", None)+ else:+ sys.modules["huggingface_hub"] = saved+ assert hub.calls == ["a/b", "a/b"], hub.calls+++def test_a_repos_allow_patterns_reach_the_hub_and_a_bare_repo_stays_unfiltered():+ """Computing the right glob and not passing it looks identical everywhere.++ The patterns are worked out in the studio builder, carried through+ `_normalise`, interpolated into generated source and echoed into the+ summary. Every one of those steps can be right while the `snapshot_download`+ call omits the keyword, and the only symptom is the 69.1 GB bill this was+ written to stop -- the summary would still print the glob it meant to use.++ The bare-string case is asserted alongside, because "filter everything"+ breaks the opposite way: a small model whose every file is loaded must not+ quietly acquire a filter and arrive incomplete.+ """+ prefetch = build_kernel._prefetch_builder()+ hub = _HubStub(hold = 0.0)+ saved = sys.modules.get("huggingface_hub")+ sys.modules["huggingface_hub"] = hub+ try:+ source = prefetch.prefetch_cell(+ [("big/gguf", ["*UD-Q4_K_XL*"]), "small/model"],+ attempt_timeout = 2,+ total_timeout = 5,+ )+ exec(compile(source, "<prefetch>", "exec"), {"__name__": "prefetch"})+ finally:+ if saved is None:+ sys.modules.pop("huggingface_hub", None)+ else:+ sys.modules["huggingface_hub"] = saved++ assert hub.calls == ["big/gguf", "small/model"], hub.calls+ assert hub.patterns_at_call == [["*UD-Q4_K_XL*"], None], hub.patterns_at_call+++def test_the_last_prefetch_attempt_falls_back_to_classic_http():+ """Retrying a STALLING transport is how a retry loop eats the session.++ Xet retries 408/429/5xx itself with backoff (5 attempts, 3s base, a+ six-minute cap per delay), so a throttled transfer can sit inside one call+ for minutes without raising. Repeating the same transport inherits that.+ The escalation to HF_HUB_DISABLE_XET is what makes the last attempt a+ genuinely different thing to try.+ """+ prefetch = build_kernel._prefetch_builder()+ seen: list = []++ class _Recording(_HubStub):+ def snapshot_download(+ self,+ repo_id = None,+ **kw,+ ):+ seen.append(os.environ.get("HF_HUB_DISABLE_XET"))+ raise RuntimeError("always")++ saved = sys.modules.get("huggingface_hub")+ before = os.environ.get("HF_HUB_DISABLE_XET")+ sys.modules["huggingface_hub"] = _Recording(hold = 0.0)+ try:+ source = prefetch.prefetch_cell(["a/b"], attempt_timeout = 1, total_timeout = 30)+ exec(compile(source, "<prefetch>", "exec"), {"__name__": "prefetch"})+ finally:+ if saved is None:+ sys.modules.pop("huggingface_hub", None)+ else:+ sys.modules["huggingface_hub"] = saved+ assert seen[-1] == "1", seen+ assert seen[:-1] == [None] * (len(seen) - 1), seen+ # ...and it is not left set for whatever runs next in this interpreter.+ assert os.environ.get("HF_HUB_DISABLE_XET") == before+++def test_the_studio_prefetch_lands_in_studios_own_cache():+ """Studio keeps a private HF_HOME, and the prefetch must follow it there.++ The t4 lane deliberately does the opposite -- it leaves HF_HOME alone so+ the training legs can read what it warms -- so the two are easy to conflate+ and the failure is silent either way: bytes land somewhere real, the+ download reports success, and the payload that wanted them downloads again.++ The install cell passes hf_home=None to INHERIT, which is only correct+ because the setup cell has already exported Studio's root. That ordering is+ what is pinned here; a prefetch cell hoisted above setup would inherit the+ image default and quietly stop helping.+ """+ studio = build_kernel._studio_builder()+ notebook = studio.build_payload_notebook(+ unsloth_ref = "x",+ repo_url = "https://h/r",+ payload_args = "--max-steps 8",+ phase = "install",+ )+ sources = ["".join(cell["source"]) for cell in notebook["cells"]]+ sets_home = [i for i, src in enumerate(sources) if 'os.environ["HF_HOME"]' in src]+ prefetches = [i for i, src in enumerate(sources) if "KAGGLE_CI_PREFETCH" in src]+ assert sets_home, "the install phase never exports Studio's HF_HOME"+ assert prefetches, "the install phase carries no prefetch"+ assert min(sets_home) < min(prefetches), (+ f"HF_HOME is exported at cell {min(sets_home)} but the prefetch runs at "+ f"{min(prefetches)}, so it would warm the image default instead"+ )+ # And it must not hardcode a root of its own alongside the inherited one.+ assert "_HF_HOME = None" in sources[min(prefetches)], sources[min(prefetches)][:400]+++def test_the_studio_prefetch_follows_the_dispatched_models():+ """--chat-model and --train-model are dispatch inputs.++ A hardcoded pair here would prefetch the defaults while the payload loaded+ something else -- which downloads happily, warms a cache nobody reads, and+ reports success.++ --chat-variant is read for the same reason and now matters as much. Studio+ loads ONE quant from a GGUF repo that ships many, so an unfiltered snapshot+ is not merely generous: run 32667451396 pulled 69.1 GB of Qwen3.5-2B-GGUF+ to serve a single UD-Q4_K_XL file, and on a 4-core Kaggle box that CPU came+ straight out of the payloads the prefetch exists to speed up.+ """+ studio = build_kernel._studio_builder()+ chat, train = studio._models_from("--chat-model a/b --train-model c/d")+ assert chat == ("a/b", ["*UD-Q4_K_XL*"]), chat+ assert train == "c/d", train+ assert studio._models_from("--chat-model=e/f")[0][0] == "e/f"++ # The filter follows the dispatched variant rather than the default, or a+ # run that overrode it would prefetch a quant it never loads.+ picked, patterns = studio._models_from("--chat-variant Q8_0")[0]+ assert patterns == ["*Q8_0*"], patterns++ # Loose at BOTH ends on purpose. A split GGUF is named+ # `...UD-Q4_K_XL-00001-of-00002.gguf`, so a suffix-anchored glob would+ # match the single-file case and miss every shard of the split one --+ # downloading nothing, reporting success, leaving Studio to fetch it.+ assert patterns[0].startswith("*") and patterns[0].endswith("*"), patterns++ defaults = studio._models_from("--max-steps 8")+ flat = [entry[0] if isinstance(entry, tuple) else entry for entry in defaults]+ payload = (SMOKE_DIR.parent / "studio_gpu" / "run_studio_gpu.py").read_text(encoding = "utf-8")+ for flag in ("--chat-model", "--train-model", "--chat-variant"):+ declared = re.search(rf'ap\.add_argument\("{flag}", default = "([^"]+)"\)', payload)+ assert declared, f"{flag} default not found in run_studio_gpu.py"+ if flag == "--chat-variant":+ assert defaults[0][1] == [f"*{declared.group(1)}*"], defaults[0]+ else:+ assert declared.group(1) in flat, (declared.group(1), flat)+++def test_the_report_shows_what_the_prefetch_achieved(tmp_path):+ """The number the leg order is arranged around has to be readable without+ downloading an artifact -- including when it says the lane did not help."""+ import report as t4_report++ evidence = tmp_path / "evidence"+ evidence.mkdir()+ (evidence / "kernel.log").write_text(+ 'KAGGLE_CI_PREFETCH {"repo": "unsloth/gpt-oss-20b", "ok": true, "seconds": 141.0, '+ '"download_seconds": 141.0, "bytes": 12000000000, "mb_per_s": 85.1, '+ '"transport": "auto", "attempts": 1}\n'+ 'KAGGLE_CI_PREFETCH {"repo": "unsloth/Qwen2.5-0.5B-Instruct", "ok": false, '+ '"seconds": 9.0, "download_seconds": null, "bytes": 0, "mb_per_s": null, '+ '"transport": "http", "attempts": 3, "error": "nope"}\n',+ encoding = "utf-8",+ )+ lines = "\n".join(t4_report.prefetch_table(evidence))+ assert "unsloth/gpt-oss-20b" in lines+ assert "141.0" in lines and "85.1" in lines and "12.0" in lines+ assert "**NO**" in lines, "a failed prefetch must be visible, not rounded away"+ assert "fallback" in lines, "a failed prefetch must say what it costs the schedule"+ # A kernel built without the lane gets no section at all, rather than a+ # table of zeroes that reads like a lane that ran and achieved nothing.+ bare = tmp_path / "bare"+ bare.mkdir()+ (bare / "kernel.log").write_text("nothing to see", encoding = "utf-8")+ assert t4_report.prefetch_table(bare) == []++ # ...and it is WIRED IN. Calling the renderer directly proves it renders,+ # which is not the same claim: deleting the one line that appends it to the+ # summary left this test green, because a table nobody calls still formats+ # perfectly. So drive main() and read what a human would actually see.+ (evidence / "launch_result.json").write_text(+ json.dumps(+ {+ "verdict": "pass",+ "reason": "all 1 payload(s) passed",+ "slug": "u/s",+ "kernel_state": "COMPLETE",+ "reports": [{"label": "control", "passed": True, "steps": []}],+ }+ ),+ encoding = "utf-8",+ )+ summary = tmp_path / "summary.md"+ proc = subprocess.run(+ [sys.executable, str(CI_DIR / "report.py"), "--evidence", str(evidence), "--expect", "1"],+ capture_output = True,+ text = True,+ env = {**os.environ, "GITHUB_STEP_SUMMARY": str(summary)},+ )+ assert proc.returncode == 0, proc.stdout+ rendered = summary.read_text(encoding = "utf-8")+ assert "model prefetch" in rendered, rendered+ assert "unsloth/gpt-oss-20b" in rendered, rendered+++def test_gptoss_never_shares_a_card(tmp_path):+ """12.78 GB of a 14.56 GB card, so it is alone by arithmetic.++ Not by a special case -- there is no `if name == "gptoss"` anywhere. If a+ leg's appetite ever grows past the budget it stops sharing on its own, and+ if gptoss ever shrinks it starts sharing on its own. What must never happen+ is the pairing that put 13.48 GB on a card and came back as an OOM reading+ like a code failure.+ """+ durations = {f"t4_{n}.ipynb": 0.4 for n in ALL_LEGS}+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2, durations = durations)+ for card, together in driven["stub"].same_card_overlaps:+ assert "t4_gptoss.ipynb" not in together, (card, together)+++def test_two_small_legs_do_share_a_card(tmp_path):+ """The feature, asserted positively.++ Every other guard here is a bound -- never over budget, never more than two+ -- and every one of them is satisfied by a scheduler that co-schedules+ NOTHING. Without this the whole change could silently do no work at all and+ the suite would stay green.+ """+ durations = {f"t4_{n}.ipynb": 0.4 for n in ALL_LEGS}+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2, durations = durations)+ assert driven[+ "stub"+ ].same_card_overlaps, "no card ever held two legs at once, so the VRAM budget bought nothing"+ assert max(driven["stub"].peak_card_legs.values()) == 2+ # The admission ledger has to balance. A leg that reserves and never+ # releases leaks capacity, and with four legs and four worker slots nothing+ # ever waits, so the leak is invisible until the day a fifth leg is wired+ # and one card silently stops taking work.+ assert driven["card_load"] and all(abs(v) < 1e-9 for v in driven["card_load"].values()), driven[+ "card_load"+ ]+ assert all(v == 0 for v in driven["card_count"].values()), driven["card_count"]+++def test_the_declared_vram_matches_what_the_legs_reported():+ """`Leg.vram_gb` decides who may share a card, and nothing checks it at+ runtime: a leg that under-declares gets admitted beside another and the+ contention comes back as an OOM attributed to whichever leg happened to+ allocate last.++ So the declared figures are checked against the peaks the payloads really+ reported, captured in the evidence of run 32611343797 and committed beside+ this test.+ """+ measured = json.loads(+ (Path(__file__).parent / "t4_smoke" / "measured_vram.json").read_text(encoding = "utf-8")+ )+ for name, peak in measured["peak_reserved_gb"].items():+ declared = LEGS[name].vram_gb+ assert declared >= peak, (+ f"{name} declares {declared} GB but peaked at {peak} GB, so the "+ "admission check would let something share a card with it that "+ "does not fit"+ )+ # ...and not so far above it that the budget stops admitting anything.+ assert declared <= peak + 1.5, (name, declared, peak)+ assert measured["card_total_gb"] > 13.0, measured+++# ------------------------------------------------- Studio sharing a card+++def test_studio_waits_for_the_queue_by_default(tmp_path, monkeypatch):+ """The default keeps both T4s visible to Studio.++ Sharing is faster and narrower, so it must be something someone turned on,+ not something that arrived with an unrelated change.+ """+ monkeypatch.setenv("CUDA_VISIBLE_DEVICES", AMBIENT_CUDA)+ durations = {f"t4_{n}.ipynb": 0.4 for n in ALL_LEGS}+ durations[STUDIO_INSTALL] = 0.1+ driven = _drive_packed(tmp_path, ALL_LEGS, gpus = 2, studio = STUDIO, durations = durations)+ calls = {c["notebook"]: c for c in driven["stub"].papermill}+ assert calls[STUDIO_TEST]["cuda"] == AMBIENT_CUDA, calls[STUDIO_TEST]+ legs = [+ (c["notebook"], c["cuda"])+ for c in driven["stub"].papermill+ if c["notebook"].startswith("t4_")+ ]+ assert len(legs) == len(ALL_LEGS)+++def test_studio_concurrent_takes_a_card_gptoss_is_not_on(tmp_path):+ """--studio-concurrent trades coverage for time and must pay honestly.++ Two things have to hold. Studio is PINNED to a card, because sharing means+ it is no longer choosing between two. And it is admitted by the same VRAM+ check the legs use, so it can never land beside gptoss: 12.78 + 2.2 is+ 14.98 on a card budgeted to 13.0, which is the pairing that came back as an+ OOM reading like a code failure.++ Driven with gptoss as the ONLY leg, so the placement is deterministic+ rather than a race between the stub's durations. An earlier version used+ all four legs and asserted on recorded overlaps; the small legs finished+ while Studio was still building its venv, so no overlap was ever recorded+ and deleting the VRAM check left the test green.+ """+ driven = _drive_packed(+ tmp_path,+ ["gptoss"],+ gpus = 2,+ studio = STUDIO,+ durations = {"t4_gptoss.ipynb": 4.0, STUDIO_INSTALL: 0.05},+ after_gpu_concurrent = True,+ )+ calls = {c["notebook"]: c for c in driven["stub"].papermill}+ assert STUDIO_TEST in calls, sorted(calls)+ assert calls[STUDIO_TEST]["cuda"] in ("0", "1"), calls[STUDIO_TEST]+ assert calls["t4_gptoss.ipynb"]["cuda"] in ("0", "1"), calls["t4_gptoss.ipynb"]+ assert calls[STUDIO_TEST]["cuda"] != calls["t4_gptoss.ipynb"]["cuda"], (+ f"Studio was put on the same card as gptoss "+ f"({calls[STUDIO_TEST]['cuda']}): 12.78 + 2.2 GB on a 13.0 GB budget"+ )+ for card, peak in driven["stub"].peak_card_gb.items():+ assert peak <= 13.0, (card, peak, driven["stub"].same_card_overlaps)+++def test_studio_concurrent_still_skips_when_its_install_failed(tmp_path):+ """The dependency survives the faster path.++ Running the assertions against a half-built tree fails on a missing venv,+ which reads like the code under test broke rather than like the install+ did -- and on this path the test half is started from its own thread, so+ the gate had to be re-implemented rather than inherited.+ """++ class _InstallFails(_PackedStub):+ def run(self, cmd, **kw):+ cmd = [str(c) for c in cmd]+ if "papermill" in cmd and STUDIO_INSTALL in " ".join(cmd):+ self.papermill.append(+ {+ "notebook": STUDIO_INSTALL,+ "cuda": None,+ "kernel": None,+ "compile_location": None,+ }+ )+ Path(cmd[cmd.index("papermill") + 2]).write_text("{}", encoding = "utf-8")+ return types.SimpleNamespace(returncode = 1, stdout = "", stderr = "")+ return super().run(cmd, **kw)++ driver = build_kernel.build_kernel(+ SMOKE_DIR,+ ALL_LEGS,+ unsloth_ref = "main",+ zoo_ref = "main",+ extra_args = (),+ per_run_timeout = 60,+ skip_reference = True,+ studio = STUDIO,+ after_gpu_concurrent = True,+ )+ stub = _InstallFails(gpus = 2, durations = {f"t4_{n}.ipynb": 0.3 for n in ALL_LEGS})+ stub.root = tmp_path+ stub.venv_root = tmp_path / "venvs"+ saved = sys.modules["subprocess"]+ sys.modules["subprocess"] = stub+ namespace: dict = {}+ try:+ for cell in driver["cells"][:2]:+ source = (+ "".join(cell["source"])+ .replace("/tmp/t4ci_venvs", str(tmp_path / "venvs"))+ .replace("/kaggle/working", str(tmp_path))+ )+ exec(compile(source, "<driver-cell>", "exec"), namespace)+ finally:+ sys.modules["subprocess"] = saved+ results = namespace.get("results") or {}+ assert STUDIO_TEST in results, sorted(results)+ assert results[STUDIO_TEST]["returncode"] is None, results[STUDIO_TEST]+ assert "install lane did not succeed" in results[STUDIO_TEST]["error"]+ assert STUDIO_TEST not in [c["notebook"] for c in stub.papermill]+++def test_a_legs_overlay_reaches_its_payload_and_never_carries_torch(tmp_path):+ """The overlay must WIN over the venv, and must not bring native packages.++ Two failures are being guarded, and they look identical from outside:++ * An overlay built but never put on ``PYTHONPATH``. The leg runs on the base+ versions, trains, passes, and reports a version table nobody reads. This+ is the reason the payload's env is inspected rather than the fact that a+ ``pip install --target`` happened.+ * An overlay that shadows torch. ``pip install --dry-run --report`` resolves+ the FULL closure, and a closure containing transformers frequently+ contains torch too; installing that into the overlay puts a second torch+ ahead of the one already loaded against this box's CUDA runtime. The stub+ resolver therefore returns torch on purpose, so a driver that forgot to+ filter fails here instead of on a Kaggle session.++ Measured basis for the mechanism: kernel unsloth-probe-overlay-t4-r2-38ac4d+ on a real T4 resolved transformers==4.57.6 + trl~=0.22.0 to three packages,+ 115.9 MB, in 10.0s, with transformers and trl imported from the overlay and+ torch still from the base.+ """+ leg = "canary"+ overlay = ("transformers==4.57.6", "trl~=0.22.0")+ original = LEGS[leg].overlay+ object.__setattr__(LEGS[leg], "overlay", overlay)+ try:+ stub = _drive_packed(tmp_path, [leg], gpus = 2)["stub"]+ finally:+ object.__setattr__(LEGS[leg], "overlay", original)++ installs = [c for c in stub.overlay_installs if "--target" in c]+ assert installs, "the leg declared an overlay and nothing was installed into one"+ target = installs[0][installs[0].index("--target") + 1]+ assert f"overlay_t4_{leg}" in target, target++ installed = " ".join(installs[0]).lower()+ assert "transformers==4.57.6" in installed, installed+ assert (+ "torch==" not in installed+ ), f"the overlay installed torch, which shadows the base one: {installed}"++ record = [p for p in stub.papermill if p["notebook"] == f"t4_{leg}.ipynb"]+ assert record, [p["notebook"] for p in stub.papermill]+ pythonpath = record[0]["env"].get("PYTHONPATH", "")+ assert target in pythonpath.split(os.pathsep), (+ f"the overlay was built at {target} but the payload's PYTHONPATH is "+ f"{pythonpath!r}, so the child would import the base versions"+ )+++def test_a_leg_with_no_overlay_gets_no_pythonpath(tmp_path):+ """The control case, without which the test above proves only that a+ variable exists somewhere.++ A driver that unconditionally set PYTHONPATH -- to the overlay root, to an+ empty directory, to anything -- would satisfy the first guard while giving+ every leg the same environment. The legs' whole purpose is that they differ.+ """+ stub = _drive_packed(tmp_path, ["control"], gpus = 2)["stub"]+ assert not [+ c for c in stub.overlay_installs if "--target" in c+ ], "a leg declaring no overlay had one built for it"+ record = [p for p in stub.papermill if p["notebook"] == "t4_control.ipynb"]+ assert record+ assert "overlay_" not in record[0]["env"].get("PYTHONPATH", "")+++def test_every_leg_installs_bitsandbytes_and_probes_that_it_imports():+ """bitsandbytes has to be asked for, and asked for EARLY.++ It is absent from every dependency set the CI resolves: `unsloth_zoo`+ declares 57 requirements and bitsandbytes is not among them, and git-main+ `unsloth` declares only seven unconditional dependencies (typer, rich,+ pydantic, pyyaml, nest-asyncio, structlog, click) with bitsandbytes reachable+ only through its CUDA extras. The released PyPI package DOES carry it+ unconditionally, which is why notebooks installing from PyPI never notice --+ and why this CI, which installs from git SHAs, must ask.++ Without it the run gets a long way before failing: the install succeeds, the+ model downloads, and it dies inside `from_pretrained` at+ unsloth_zoo/patching_utils.py:386. Probing it in the import cell turns that+ into a failure before the session is spent, which is the whole point of the+ probe list.+ """+ for name, leg in LEGS.items():+ flat = [spec for group in leg.install for spec in group]+ assert any("bitsandbytes" in spec for spec in flat), (+ f"leg {name!r} never installs bitsandbytes; on the Kaggle image it "+ f"would fail inside from_pretrained after the model download"+ )+ assert "bitsandbytes" in leg.imports, (+ f"leg {name!r} does not probe bitsandbytes, so a broken or missing "+ f"copy surfaces minutes later as a model-loading error"+ )diff --git a/tests/kaggle/test_t4_payload_assertions.py b/tests/kaggle/test_t4_payload_assertions.pyindex 2532b9fb086..4c3629e45cc 100644--- a/tests/kaggle/test_t4_payload_assertions.py+++ b/tests/kaggle/test_t4_payload_assertions.py@@ -15,6 +15,7 @@ costs a Kaggle session has to be checkable without one.from __future__ import annotationsimport json+import reimport sysimport typesfrom pathlib import Path@@ -908,6 +909,26 @@ def test_the_committed_reference_pins_the_model_it_was_captured_on():# -------------------------------------------- run_t4_smoke.py: main() paths+def _batch_record(**over):+ """A healthy batched-generation record, with fields overridable per test."""+ base = {+ "prompt_token_lengths": [11, 14, 9, 17, 12, 20, 8, 15],+ "distinct_lengths": 8,+ "padding_side_observed": "left",+ "padding_side_after": "left",+ "singles": [f"out{i}" for i in range(8)],+ "batched": {+ "2": [f"out{i}" for i in range(8)],+ "4": [f"out{i}" for i in range(8)],+ "8": [f"out{i}" for i in range(8)],+ },+ "agrees": {"2": True, "4": True, "8": True},+ "empty_outputs": [],+ }+ base.update(over)+ return base++def _cycle(index: int, losses: list[float]) -> dict:return {"run_index": index,@@ -916,6 +937,13 @@ def _cycle(index: int, losses: list[float]) -> dict:],"generated": "__UNSLOTH__!!!","canary_found": True,+ # A simulated HEALTHY cycle has to look healthy in every respect a real+ # one is judged on, this record included. Leaving it out made+ # `--check-batched-generation` (on by default, because a leg that+ # quietly skips it stops covering #3699/#1066/#1456/#2138) fail every+ # simulated run with "batched generation was never run" -- which is the+ # rule working, on a fixture that had not kept up.+ "batched_generation": _batch_record(),"adapter_files": ["adapter_config.json", "adapter_model.safetensors"],"saved_adapter": _adapter_state(),"determinism": {},@@ -1773,3 +1801,569 @@ def test_gptoss_still_reads_a_bf16_card_and_a_t4_the_way_it_did():)== [])+++def test_batched_generation_runs_end_to_end_against_a_stub_model():+ """Every other test in this section feeds `batched_generation_failures` a+ dict someone typed. That checks the RULE and never once executes the code+ that produces the dict, which is how kernel unsloth-probe-defaultleg-723c28+ trained all ten steps on a real T4 and then died on++ NameError: name 'torch' is not defined++ inside `batched_generation` itself. Every torch user in run_t4_smoke.py+ imports it inside the function; that one did not, and no CPU test noticed+ because none of them ever called it.++ So drive the real function with a stub tokenizer and model. The stub echoes+ a deterministic continuation per row, so agreement across batch sizes is+ guaranteed and this asserts the plumbing rather than the model: shapes,+ the padded-width slice, and that the function runs at all.+ """+ import torch++ from run_t4_smoke import batched_generation, batched_generation_failures++ class _Enc(dict):+ def to(self, _device):+ return self++ class _Tok:+ padding_side = "right"+ pad_token = "<pad>"+ eos_token = "<eos>"+ pad_token_id = 0++ def __call__(+ self,+ text,+ return_tensors = None,+ padding = False,+ ):+ texts = [text] if isinstance(text, str) else list(text)+ # One token per character, so a length spread in the prompts is a+ # length spread in the ids and the padding is real.+ ids = [[ord(c) % 100 + 1 for c in t] for t in texts]+ if return_tensors is None:+ return {"input_ids": ids[0] if isinstance(text, str) else ids}+ width = max(len(i) for i in ids)+ # LEFT padding, which is what the function asks for and what the+ # padded-width slice below depends on.+ padded = [[0] * (width - len(i)) + i for i in ids]+ return _Enc(+ input_ids = torch.tensor(padded),+ attention_mask = torch.tensor([[0] * (width - len(i)) + [1] * len(i) for i in ids]),+ )++ def decode(+ self,+ row,+ skip_special_tokens = False,+ ):+ return "".join(chr(int(v)) for v in row if int(v) != 0)++ class _Model:+ device = "cpu"++ def generate(+ self,+ input_ids = None,+ attention_mask = None,+ max_new_tokens = 8,+ **_kw,+ ):+ # Append the same continuation to every row, derived from that+ # row's own unpadded content, so batching cannot change it.+ outs = []+ for row in input_ids:+ real = [int(v) for v in row if int(v) != 0]+ tail = [(sum(real) % 26) + 65] * max_new_tokens+ outs.append([int(v) for v in row] + tail)+ return torch.tensor(outs)++ # Eight, because BATCH_SIZES tops out at 8 and the rule rejects a record+ # whose largest batch could never have been formed. Lengths 1..8 so every+ # batch pads and the padded-width slice is exercised rather than skipped.+ prompts = ["abcdefgh"[:n] * 1 for n in range(1, 9)]+ record = batched_generation(_Model(), _Tok(), prompts, max_new_tokens = 4)++ assert record["distinct_lengths"] == 8, "the stub prompts must actually pad"+ assert record["padding_side_observed"] == "left"+ assert len(record["singles"]) == len(prompts)+ assert all(s for s in record["singles"]), "the padded-width slice dropped everything"+ assert batched_generation_failures(record) == []+++def test_a_healthy_batched_generation_record_reports_no_failures():+ from run_t4_smoke import batched_generation_failures+ assert batched_generation_failures(_batch_record()) == []+++def test_a_batch_whose_prompts_are_all_one_length_is_reported_as_proving_nothing():+ from run_t4_smoke import batched_generation_failures++ """The vacuity case, and the reason this check exists in this shape.++ Left padding only happens when the prompts in a batch differ in length. Feed+ eight identical-length prompts and every batch agrees with every single --+ perfectly, every time, having never padded once. That is a green+ left-padding assertion covering nothing, which is the failure mode this+ whole file keeps being caught by.+ """+ failures = batched_generation_failures(+ _batch_record(prompt_token_lengths = [12] * 8, distinct_lengths = 1)+ )+ assert any("nothing was ever padded" in f for f in failures), failures+++def test_a_padding_side_silently_flipped_to_right_is_a_failure():+ from run_t4_smoke import batched_generation_failures++ """#2138: a release forced the tokenizer padding side to right in inference.++ Checked twice, before and after generating, because the override that+ caused it happened INSIDE the inference path -- so the value this payload+ set is not evidence of the value that was used.+ """+ assert any(+ "padding_side_after" in f+ for f in batched_generation_failures(_batch_record(padding_side_after = "right"))+ )+ assert any(+ "padding_side_observed" in f+ for f in batched_generation_failures(_batch_record(padding_side_observed = "right"))+ )+++def test_a_batch_that_disagrees_with_one_at_a_time_is_a_failure():+ from run_t4_smoke import batched_generation_failures++ """#3699 / #1456: batched output diverging from sequential greedy output."""+ broken = _batch_record()+ broken["batched"]["4"] = ["different"] * 8+ broken["agrees"]["4"] = False+ failures = batched_generation_failures(broken)+ assert any("batch size 4 did not reproduce" in f for f in failures), failures+++def test_empty_generations_are_a_failure_even_when_every_batch_agrees():+ from run_t4_smoke import batched_generation_failures++ """#1066: gibberish, of which "nothing at all" is the degenerate case.++ Agreement alone is not health: a model that emits an empty string for every+ prompt agrees with itself at every batch size.+ """+ failures = batched_generation_failures(+ _batch_record(+ singles = [""] * 8,+ empty_outputs = [0, 1, 2, 3, 4, 5, 6, 7],+ batched = {"2": [""] * 8, "4": [""] * 8, "8": [""] * 8},+ )+ )+ assert any("generated nothing at all" in f for f in failures), failures+++def test_too_few_prompts_to_fill_the_largest_batch_is_reported():+ from run_t4_smoke import batched_generation_failures++ """A batch size larger than the prompt list silently becomes one small batch."""+ failures = batched_generation_failures(+ _batch_record(singles = ["a", "b"], prompt_token_lengths = [5, 9], distinct_lengths = 2)+ )+ assert any("never actually formed" in f for f in failures), failures+++def test_a_missing_batched_record_is_a_failure_not_a_pass():+ from run_t4_smoke import batched_generation_failures+ """A leg where the check never ran must not look like a leg where it passed."""+ assert batched_generation_failures(None) == ["batched generation was never run"]+++# ------------------------------------------------- run_grpo_t4.py: the reward+++def _completions(*lengths):+ """GRPO hands reward functions a list of message lists."""+ return [[{"role": "assistant", "content": "x" * n}] for n in lengths]+++def test_the_length_reward_still_discriminates_at_the_lengths_the_model_emits():+ """The leg's only instrument, and it was broken in the least visible way.++ `reward_length` was `min(len(t), 200) / 200.0` while its docstring claimed+ to be "SENSITIVE to a group's diversity". Kernels+ unsloth-probe-grpo-rep2-b03be8 and -rep3-bc3828 recorded completions of+ 2534 to 3396 characters, so every completion scored exactly 1.0, every+ group tied, and the leg failed with `reward_std was zero on every step`.+ Two runs in three died on it.++ A reward that saturates below the range the model actually occupies reads+ as a broken generation path in the report. So assert discrimination at the+ OBSERVED lengths, not at convenient small ones: the old function passes any+ test written with 10- and 20-character completions.+ """+ sys.path.insert(0, str(SMOKE_DIR))+ from run_grpo_t4 import reward_length++ for a, b in ((2534, 3396), (3013, 3328), (2738, 2633)):+ scores = reward_length(_completions(a, b))+ assert scores[0] != scores[1], (+ f"lengths {a} and {b} scored identically ({scores}); the reward has "+ f"saturated in the range the model actually produces"+ )+++def test_the_length_reward_is_strictly_increasing_and_never_saturates():+ """The property that makes the test above hold for lengths nobody has seen+ yet. A cap is a length at which this stops being true, which is exactly how+ the previous version passed review."""+ from run_grpo_t4 import reward_length++ lengths = [0, 1, 50, 200, 1000, 3000, 10000, 100000]+ scores = reward_length(_completions(*lengths))+ assert all(x < y for x, y in zip(scores, scores[1:])), scores+ assert all(0.0 <= s < 1.0 for s in scores), scores+++def test_two_identical_completions_still_tie_because_that_is_real():+ """Guard against overcorrecting. A group whose completions are genuinely+ identical SHOULD tie -- that is degenerate generation and the leg is right+ to fail on it. The bug was ties between DIFFERENT completions."""+ from run_grpo_t4 import reward_length++ scores = reward_length(_completions(2534, 2534))+ assert scores[0] == scores[1]+++# --------------------------------------------------- gguf_export.py+++def _gguf_record(**over):+ """A healthy export, shaped like the one measured on+ unsloth-probe-gguf-q8-peft-920e3e: the GGUF in the SIBLING directory."""+ record = {+ "save_dir": "/tmp/q8p",+ "requested_quantization": "q8_0",+ "ok": True,+ "seconds": 40.6,+ "ggufs": [+ {+ "path": "/tmp/q8p_gguf/qwen3-0.6b.Q8_0.gguf",+ "mb": 609.8,+ "found_in": "/tmp/q8p_gguf",+ "suffix": "_gguf",+ }+ ],+ }+ record.update(over)+ return record+++def test_a_healthy_gguf_export_reports_no_failures():+ from gguf_export import export_failures+ assert export_failures(_gguf_record(), accept_quantizations = ("q8_0",)) == []+++def test_an_export_that_reported_ok_but_wrote_no_gguf_is_a_failure():+ """The trap this module exists for. save_pretrained_gguf writes the merged+ safetensors into the directory it was given and the GGUF into a SIBLING, so+ code that globs the directory it passed finds nothing, raises nothing, and+ calls it a successful export."""+ from gguf_export import export_failures++ failures = export_failures(_gguf_record(ggufs = []), accept_quantizations = ("q8_0",))+ assert failures and "no .gguf" in failures[0]+ assert "_gguf sibling" in failures[0]+++def test_a_gguf_search_finds_the_file_in_the_sibling_directory(tmp_path):+ """Executed against a real directory pair rather than a hand-written dict,+ because the whole point is WHERE the file is."""+ from gguf_export import find_ggufs++ save = tmp_path / "out"+ save.mkdir()+ (save / "model.safetensors").write_bytes(b"0" * 2048)+ sibling = tmp_path / "out_gguf"+ sibling.mkdir()+ (sibling / "qwen3-0.6b.Q8_0.gguf").write_bytes(b"0" * 4096)++ found = find_ggufs(str(save))+ assert len(found) == 1, found+ assert found[0]["suffix"] == "_gguf"+ assert found[0]["path"].endswith("qwen3-0.6b.Q8_0.gguf")+++def test_a_gguf_that_is_only_a_header_is_not_an_export():+ from gguf_export import export_failures+ failures = export_failures(+ _gguf_record(+ ggufs = [+ {+ "path": "/tmp/q8p_gguf/x.Q8_0.gguf",+ "mb": 0.1,+ "found_in": "/tmp/q8p_gguf",+ "suffix": "_gguf",+ }+ ]+ ),+ accept_quantizations = ("q8_0",),+ )+ assert failures and "header and no weights" in failures[0]+++def test_a_model_allowed_to_override_the_quantization_still_passes():+ """gpt-oss answers q8_0 with "GPT-OSS does not support GGUF quantization+ (requested: q8_0). Overriding to MXFP4 format." That is documented+ behaviour, so a leg that accepts MXFP4 must not fail on it."""+ from gguf_export import export_failures++ record = _gguf_record(+ ggufs = [+ {+ "path": "/tmp/g_gguf/gpt-oss-20b.MXFP4.gguf",+ "mb": 11800.0,+ "found_in": "/tmp/g_gguf",+ "suffix": "_gguf",+ }+ ]+ )+ assert export_failures(record, accept_quantizations = ("mxfp4",)) == []+ # ... and a leg that does NOT accept it still says so.+ failures = export_failures(record, accept_quantizations = ("q8_0",))+ assert failures and "accepted quantization" in failures[0]+++def test_a_gguf_that_no_runner_could_execute_is_a_failure():+ from gguf_export import run_failures+ failures = run_failures(+ {+ "gguf": "/tmp/q8p_gguf/x.gguf",+ "bench": {"seconds": 240.0, "error": "TimeoutExpired: ..."},+ "completion": {"seconds": 240.0, "returncode": 1, "stderr": "bad magic"},+ }+ )+ assert failures and "produced no output from any runner" in failures[0]+++def test_one_successful_runner_is_enough():+ from gguf_export import run_failures+ assert (+ run_failures(+ {+ "gguf": "/tmp/q8p_gguf/x.gguf",+ "bench": {"seconds": 12.0, "returncode": 0, "stdout": "tg128 ... 41.2"},+ }+ )+ == []+ )+++def test_a_bundle_with_no_runners_at_all_is_reported_as_that():+ """Distinct from "the file does not run": a missing binary is a bundle+ problem and blaming the model file for it would send the reader to the+ wrong place."""+ from gguf_export import run_failures++ failures = run_failures(+ {+ "gguf": "/tmp/x.gguf",+ "bench": {"skipped": "no llama-bench in the bundle"},+ "completion": {"skipped": "no llama-completion in the bundle"},+ }+ )+ assert failures and "missing from the llama.cpp bundle" in failures[0]+++def test_the_llama_cpp_facts_read_a_tuple_not_a_directory(tmp_path):+ """install_llama_cpp returns (llama-quantize, convert_hf_to_gguf.py). An+ earlier probe treated the return value as a bin directory and reported+ "0 binaries", which was the probe being wrong, not the bundle being+ empty."""+ from gguf_export import llama_cpp_facts++ quant = tmp_path / "llama-quantize"+ conv = tmp_path / "convert_hf_to_gguf.py"+ quant.write_text("")+ conv.write_text("")++ facts = llama_cpp_facts(+ "Unsloth: Installing prebuilt llama.cpp b10472-mix-4b653db "+ "(app-b10472-mix-4b653db-linux-x64-cpu.tar.gz) - skipping compilation.",+ (str(quant), str(conv)),+ )+ assert facts["all_exist"] is True+ assert facts["dir"] == str(tmp_path)+ assert facts["prebuilt"] is True+ assert facts["source_build_markers"] == []+++def test_a_source_build_is_visible_even_though_it_succeeds():+ """The failure to catch is not "kernel missing" but "kernel built from+ source": silent, correct, and many minutes on 4 vCPUs."""+ from gguf_export import llama_cpp_facts++ facts = llama_cpp_facts("-- Configuring done\ncmake --build . -j 4\n", ())+ assert facts["prebuilt"] is False+ assert "cmake" in facts["source_build_markers"]+++# ------------------------------- run_t4_smoke.py: parent -> child argv+++# Options the PARENT alone acts on, so their absence from the child command is+# correct rather than a leak. Each is here for a stated reason; an entry added+# without one is how this guard stops working.+PARENT_ONLY_DESTS = {+ "outdir", # the parent gives each cycle its own subdirectory+ "cycle", # set by the parent per child, never forwarded verbatim+ "repeat", # how many children to launch+ "reference", # band check runs in the parent, over collected cycles+ "rel_tol", # ... and its tolerances+ "abs_floor",+ "require_canary", # evaluated by the parent's failure collector+ "check_batched_generation",+ "export_gguf", # forwarded as a bare flag, asserted separately below+ # The pin check reads the report the cycles produced, in the parent+ # (run_t4_smoke.py:1741), so the children have nothing to do with it.+ "pins",+ # The plain-TRL control arm is spawned BY the parent, after the cycles, and+ # ruled on there. A cycle child neither runs it nor judges it.+ "compare_naive_trl",+ "control_oom_is_ok",+ # Kernel provenance IS collected in the child (it needs the loaded model),+ # but the flag reaches it through the bare-flag block rather than the+ # name/value pairs this check walks; asserted separately in+ # test_kernel_provenance.py.+ "kernel_provenance",+ # The vision run is spawned BY the parent, after the cycles and in a+ # process of its own: it loads a second model, and two 4bit models resident+ # at once on a 14.56GB T4 is how a leg becomes an OOM blamed on the thing+ # it was testing.+ "vision_run",+}+++def _smoke_source() -> str:+ return (SMOKE_DIR / "run_t4_smoke.py").read_text(encoding = "utf-8")+++def _child_command_block() -> str:+ """The argv the parent builds for each cycle."""+ source = _smoke_source()+ start = source.index("cmd = [\n sys.executable,")+ end = source.index("proc = subprocess.run(cmd)", start)+ return source[start:end]+++def test_every_option_the_child_needs_actually_reaches_the_child():+ """The class of bug, not one instance of it.++ Cycles run in fresh child processes and the parent rebuilds their argv from+ an explicit list. A flag added to the parser but not to that list is+ accepted on the command line, parsed, logged in the driver's exec line, and+ silently ignored -- which is exactly what happened to --export-gguf on+ kernel unsloth-probe-default-gguf-637565: the leg failed with "GGUF export+ was never run" while the driver log showed --export-gguf right there in the+ command.++ --check-batched-generation escaped this only because it defaults to True, so+ the child got it without being told. That is luck, not design.+ """+ import argparse+ import importlib++ sys.path.insert(0, str(SMOKE_DIR))+ module = importlib.import_module("run_t4_smoke")++ # Build the parser the same way main() does, by calling it with a sentinel+ # that makes it return rather than run.+ parser = argparse.ArgumentParser()+ source = _smoke_source()+ dests = set(re.findall(r'dest\s*=\s*"([a-z_0-9]+)"', source))+ dests |= {+ m.replace("-", "_") for m in re.findall(r'ap\.add_argument\(\s*"--([a-z0-9-]+)"', source)+ }+ assert "export_gguf" in dests, "the parser no longer defines --export-gguf"+ assert "model" in dests, "the dest scrape found nothing; fix the scrape"++ block = _child_command_block()+ forwarded = {m.replace("-", "_") for m in re.findall(r'"--([a-z0-9-]+)"', block)}++ missing = sorted(d for d in dests - forwarded - PARENT_ONLY_DESTS if not d.startswith("no_"))+ assert not missing, (+ f"these options are parsed but never forwarded to the cycle child, so "+ f"setting them does nothing: {missing}. Either forward them or list "+ f"them in PARENT_ONLY_DESTS with a reason."+ )+ del parser, module+++def test_the_export_flag_is_forwarded_as_a_bare_flag():+ """store_true options cannot ride the (flag, value) loop -- `--export-gguf+ True` is not a thing -- so they need their own append, and that is the line+ that was missing."""+ block = _child_command_block()+ assert 'cmd.append("--export-gguf")' in block+ assert "if args.export_gguf:" in block+++def test_the_export_settings_ride_the_value_loop():+ block = _child_command_block()+ assert '("--gguf-quantization", args.gguf_quantization)' in block+ assert '("--gguf-accept", args.gguf_accept)' in block+++def test_a_second_cycle_cannot_report_an_already_installed_llama_cpp_as_a_source_build():+ """Measured on unsloth-probe-visleg-full-b3a317, and it is a trap.++ The prebuilt banner is printed once, by the install that downloads the+ bundle. A second cycle in the same session finds llama.cpp already there+ and prints nothing, so the field read `prebuilt: true` on cycle 0 and+ `prebuilt: false` on cycle 1 for the SAME installation -- and `false` reads+ as "built from source", which is the one thing this field exists to catch.++ None is the third state: this run did not install it, so it cannot say.+ """+ from gguf_export import llama_cpp_facts++ quiet = llama_cpp_facts("", ())+ assert (+ quiet["prebuilt"] is None+ ), "an install that printed nothing must not claim a source build"+ assert quiet["source_build_markers"] == []++ # And the two real answers are unchanged.+ assert (+ llama_cpp_facts(+ "Unsloth: Installing prebuilt llama.cpp b10472 - skipping compilation.", ()+ )["prebuilt"]+ is True+ )+ assert llama_cpp_facts("cmake --build . -j 4\n", ())["prebuilt"] is False+++def test_the_text_leg_gguf_export_does_not_land_in_the_artifact_directory():+ """`/kaggle/working` is 21.0 GB and is what `kernels output` ships back.++ Measured on unsloth-probe-lcleg-final-a90fbb, which is the run that found+ this:++ RuntimeError: Unsloth: Not enough disk space to convert to GGUF.+ The export needs about 16.6GB on the filesystem holding+ `/kaggle/working/t4_out_Latest_compile/cycle0/gguf_run0`++ Two failures in one. A merge too big for that volume kills the leg, and a+ merge that DOES fit is downloaded as part of the artifact, which nobody+ wanted. gpt-oss and the vision run were both moved to a tempdir for exactly+ this reason and this path was missed -- so a small model kept passing and+ hid it.+ """+ src = _smoke_source()+ call = src[src.index("gguf_export_record = export_gguf(") :]+ call = call[: call.index(")")]+ assert "tempfile.mkdtemp" in call, "the export writes into the artifact directory"+ assert (+ "args.outdir" not in call+ ), "args.outdir is /kaggle/working, which is 21 GB and is collected"diff --git a/tests/kaggle/test_t4_smoke_harness.py b/tests/kaggle/test_t4_smoke_harness.pyindex d8c8fe3ac72..beea60737e8 100644--- a/tests/kaggle/test_t4_smoke_harness.py+++ b/tests/kaggle/test_t4_smoke_harness.py@@ -1544,7 +1544,23 @@ def test_every_setting_the_child_needs_is_forwarded_to_it():# ------------------------------------------------------------ kernel build-LEG_NAMES = ("control", "canary", "gptoss", "grpo")+LEG_NAMES = (+ "control",+ "canary",+ "gptoss",+ "grpo",+ "default",+ # Unpinned by design (all_cards), so its generated cells differ from every+ # other leg's: the visible-GPU assertion in the preamble compares against 2.+ "multi_gpu",+ "latest_compile",+ "vision_fla_compile",+ # Retired in favour of vision_fla_compile and kept buildable: an unwired+ # leg is unwired because of its install or an open question, not because+ # the payload rots, and a retirement that stops building is one nobody can+ # reverse in the hour they need it.+ "frontier",+)def _build(@@ -1740,7 +1756,7 @@ def test_every_leg_carries_the_version_recorder(tmp_path):assert "versions.py" in COMMON_FILESfor name in LEGS:- payload = _payload_notebooks(_build(tmp_path / name, name))[f"t4_{name}.ipynb"]+ payload = _payload_notebooks(_build(tmp_path / name, name))[f"t4_{LEGS[name].name}.ipynb"]assert "versions.py" in _cell(payload, 0)assert "versions.flatten_versions" in _cell(payload, 2)@@ -1757,7 +1773,7 @@ def test_every_registered_leg_is_either_carried_or_explicitly_unwired():carried = [name for kernel in KERNELS for name in kernel]assert len(carried) == len(set(carried)), carried- assert sorted(carried) + sorted(UNWIRED) == sorted(LEGS), (+ assert sorted(carried + list(UNWIRED)) == sorted(LEGS), (sorted(carried),sorted(UNWIRED),sorted(LEGS),@@ -1810,10 +1826,12 @@ def test_generated_cells_compile(tmp_path):for index, cell in enumerate(nb["cells"]):compile("".join(cell["source"]), f"{path}/{name}#cell{index}", "exec")seen += 1- # 5 builds x (3 driver cells + 4 payload cells), asserted so a refactor that- # stops reaching the payloads cannot leave this test passing while compiling- # nothing that matters.- assert seen == 5 * 7, seen+ # One build per leg in LEG_NAMES x (3 driver cells + 4 payload cells),+ # asserted so a refactor that stops reaching the payloads cannot leave this+ # test passing while compiling nothing that matters. Derived from LEG_NAMES+ # rather than hardcoded: the literal 5 silently meant "every leg" only for+ # as long as nobody added one.+ assert seen == (len(LEG_NAMES) + 1) * 7, seendef _undefined_names(source: str, already_bound: set) -> tuple:@@ -2139,9 +2157,21 @@ def test_the_files_the_payload_carries_are_byte_identical_to_the_repo(tmp_path):"training_evidence.py","run_t4_smoke.py","determinism.py",+ "gguf_export.py",+ "naive_trl_compare.py",+ "kernel_provenance.py","pins/control.txt","references/t4_qwen2.5-0.5b.json",}, sorted(files)+ # Cross-checked against the registry as well as the literal above. The+ # literal is what makes an accidental addition visible in a diff; the+ # registry check is what stops the two drifting, which is how a leg ends up+ # shipping a file no test knows about.+ from legs import LEGS++ leg = LEGS["control"]+ declared = set(leg.files) | ({f"references/{leg.reference}"} if leg.reference else set())+ assert set(files) == declared, sorted(declared)for name, data in files.items():assert gzip.decompress(base64.b64decode(data)) == (SMOKE_DIR / name).read_bytes(), name@@ -2279,6 +2309,99 @@ def test_the_grpo_leg_names_its_attention_backend():assert LEGS["grpo"].env.get("VLLM_ATTENTION_BACKEND") == "TRITON_ATTN"+def test_the_grpo_leg_disables_flashinfer_at_the_only_layer_that_holds():+ """Two Kaggle probes died four rungs each on the same flashinfer link, one+ of them WITH `VLLM_USE_FLASHINFER_SAMPLER=0` already set, because that is+ not the layer the decision is made at. unsloth_zoo's patch_vllm assigns+ both vLLM env vars itself during model load:++ vllm_utils.py:2494 VLLM_ATTENTION_BACKEND = "FLASHINFER"+ vllm_utils.py:2502 VLLM_USE_FLASHINFER_SAMPLER = "1"++ each behind `elif Version(vllm_version) >= Version("0.11.0")`, which this+ leg's 0.19.1 pin satisfies. Its guard checks only that nvcc and ninja are+ present, and on Kaggle both are - the missing piece is the driver stub the+ compiled objects link against, which that check does not look at. So the+ two settings above are overwritten before vLLM ever reads them.++ UNSLOTH_VLLM_NO_FLASHINFER is read at vllm_utils.py:2449, ahead of every+ assignment, and is the only one of the three that survives. Losing it puts+ the leg straight back on a link that cannot succeed on this image."""+ from legs import LEGS+ assert LEGS["grpo"].env.get("UNSLOTH_VLLM_NO_FLASHINFER") == "1"+++def test_the_grpo_leg_removes_flashinfer_and_the_removal_reaches_the_payload(tmp_path):+ """The env vars were necessary and NOT sufficient, established over four+ Kaggle ladder probes:++ r1 nothing set 4/4 rungs fail, cached_ops/sampling+ r2 VLLM_USE_FLASHINFER_SAMPLER=0 identical failure+ r3 UNSLOTH_VLLM_NO_FLASHINFER=1 sampling build gone; now fails in+ cached_ops/batch_prefill_with_kv_cache_...+ r4 uninstall flashinfer PASSES on the first rung, 11.30 GB++ r3 is the one that proves the remaining gap: the sampler build genuinely+ stopped, and the failure moved to ATTENTION, which is chosen by+ VLLM_ATTENTION_BACKEND -- already TRITON_ATTN and confirmed inherited by the+ rung child -- and which vLLM offers no prefill-specific opt-out for. There+ was no variable left to set, which is why the package goes instead.++ THE SHIPPED LEG FAILS DIFFERENTLY, and that is the reason this guard is+ worth keeping rather than a historical note. Removing the uninstall from the+ leg (unsloth-probe-grpo-noun-05777b, nothing else changed) does NOT+ reproduce the link error at all -- the leg carries a libcuda shim+ (run_grpo_t4.py:597, /usr/local/cuda/compat/libcuda.so, since #8440) that+ the probes lacked. It fails instead with++ AcceleratorError: CUDA error: an illegal memory access was encountered++ which is verbatim the intermittent crash UNWIRED["grpo"] documents from+ kernels unsloth-t4-ci-70a2f4eb and -c98f14be. With the uninstall the crash+ has not recurred.++ State the strength honestly: that crash is documented at roughly two failures+ in three, so a handful of clean runs is suggestive and not proof, and this+ guard asserts only that the uninstall is PRESENT and REACHES the payload --+ which removing it demonstrably breaks. It does not claim to have identified+ the crash's cause.++ Asserted through the BUILT payload rather than off the dataclass: a field+ nothing emits is a setting that reads like coverage and does nothing, and+ that is the exact failure mode this file exists to catch.+ """+ from legs import LEGS++ assert set(LEGS["grpo"].uninstall) >= {"flashinfer-python", "flashinfer-cubin"}+ payload = _payload_notebooks(_build(tmp_path / "g", "grpo"))["t4_grpo.ipynb"]+ install_cell = _cell(payload, 1)+ assert "flashinfer-python" in install_cell+ assert "uninstall" in install_cell+++def test_a_leg_with_nothing_to_uninstall_does_not_run_pip_uninstall(tmp_path):+ """The counterpart, so the emission is conditional rather than always-on:+ an unconditional `pip uninstall -y` with an empty list is a per-leg+ subprocess that can only cost time."""+ from legs import LEGS++ assert LEGS["control"].uninstall == ()+ payload = _payload_notebooks(_build(tmp_path / "c", "control"))["t4_control.ipynb"]+ assert "flashinfer" not in _cell(payload, 1)+++def test_the_grpo_leg_asks_for_the_utilization_both_platforms_measured(tmp_path):+ """0.95 is measured, not requested-and-hoped. Colab (torch 2.11.0) peaked at+ 11.76 GB and Kaggle (torch 2.10.0) at 11.30 GB, both on the FIRST rung of a+ 0.95/0.8/0.6/0.5 ladder and both surviving three sleep/wake cycles, roughly+ 3 GB under a 14.56 GB card. The 0.5 that used to be here came from Qwen3-4B+ probes and was never measured for the 0.6B this leg trains."""+ from legs import LEGS++ args = LEGS["grpo"].args+ assert args[args.index("--gpu-memory-utilization") + 1] == "0.95"++def test_the_grpo_leg_no_longer_carries_xformers():"""Its vLLM backend is gone at this version, so it would be a packagenothing selects, resolved against a torch it has opinions about."""@@ -2305,13 +2428,74 @@ def test_nothing_is_both_wired_and_unwired():assert not (wired & set(UNWIRED)), sorted(wired & set(UNWIRED))-def test_an_unwired_note_says_what_is_unknown():+def test_an_unwired_note_says_what_is_unknown_or_what_replaced_it():"""An unwired leg whose note reads as settled is a leg someone wires without- running it. Vacuous while UNWIRED is empty, and correctly so: it has- something to check the moment a leg goes back in."""- from legs import UNWIRED+ running it.++ THREE reasons a leg is not wired, and they are not the same thing.++ A leg with an open question says STILL UNKNOWN and names it.++ A leg that was REPLACED has nothing unknown about it at all -- frontier is+ retired because vision_fla_compile asserts everything it did and more --+ and forcing that note to claim an open question would be a lie in the file+ that exists to stop lies of exactly that kind.++ A leg that was MEASURED AND REJECTED is the third, added when multi_gpu hit+ it: the leg passes on hardware and was still held out, because an A/B+ showed it costing wall clock and breaking another leg. Nothing about it is+ unknown and nothing replaced it, so the first two categories would both be+ false. What that note owes the reader instead is the way back in, so it+ must say WHAT WOULD UNBLOCK IT -- otherwise "rejected" reads as permanent+ and the measurement behind it is never revisited.++ A leg that is UNDER RE-MEASUREMENT is the fourth, and it exists because a+ rejection can turn out to rest on a defect somewhere else. multi_gpu was+ rejected partly for breaking the Default leg; that break was this driver+ building the leg on the wrong python and installing pyarrow where dill+ pickles it by value, and it is fixed. Leaving the note reading REJECTED+ would keep a withdrawn measurement standing as a finding, which is the+ failure this file exists to catch, and forcing it to read STILL UNKNOWN+ would throw away everything that IS known. Such a note owes the reader the+ run that will settle it, by name, so the answer is collectable rather than+ perpetually pending.++ So the rule is: say which of the four it is, name the superseding leg if it+ was superseded, name the unblock condition if it was rejected, and name the+ deciding run if it is being re-measured. A bare note passes none of them.+ """+ from legs import LEGS, UNWIREDfor name, note in UNWIRED.items():- assert "STILL UNKNOWN" in note, f"{name} note does not say what is open"+ if "SUPERSEDED" in note:+ named = [other for other in LEGS if other != name and other in note]+ assert named, (+ f"{name} says it was superseded and does not name what by; a "+ f"retirement nobody can trace is a deletion with extra steps"+ )+ continue+ if "MEASURED AND REJECTED" in note:+ assert "WHAT WOULD UNBLOCK IT" in note, (+ f"{name} says it was measured and rejected without saying what "+ f"would change that; a rejection with no way back is a deletion "+ f"that keeps costing a reader the time to re-derive it"+ )+ continue+ if "UNDER RE-MEASUREMENT" in note:+ assert "WHAT WOULD UNBLOCK IT" in note, (+ f"{name} says it is being re-measured without saying what would "+ f"settle it, which is a rejection with the deadline removed"+ )+ # The deciding run has to be NAMED. "we will re-run it" is how a+ # measurement stays pending for a year.+ assert re.search(r"\b(ab\d+|unsloth-probe-[a-z0-9-]+)\b", note), (+ f"{name} is under re-measurement and names no run that will "+ f"decide it, so nobody can go and read the answer"+ )+ continue+ assert "STILL UNKNOWN" in note, (+ f"{name} note says none of: what is open, what replaced it, or "+ f"what would unblock it"+ )def test_grpo_stays_unwired_while_the_illegal_memory_access_is_open():@@ -2359,12 +2543,21 @@ def test_the_grpo_leg_keeps_the_config_that_actually_fit():unsloth_zoo/gradient_checkpointing.py:1013, peaking at 15.97GB in 16-bit and19.25GB in 4-bit. The set below passed on kernel unsloth-t4-ci-53efcc4e at13.60GB with reward_std 0.707 at step 2, so restoring any of them to the- notebook's value is a session that OOMs, and it fails here instead."""+ notebook's value is a session that OOMs, and it fails here instead.++ UTILIZATION IS THE ONE EXCEPTION, raised 0.5 -> 0.95 on measurement. Those+ probes trained **Qwen3-4B**; this leg trains Qwen3-0.6B, where the+ activation budget is a different problem entirely. A 0.95/0.8/0.6/0.5 ladder+ stopped at the FIRST rung on both platforms - Colab peak reserved 11.76GB,+ Kaggle 11.30GB, three sleep/wake cycles each - roughly 3GB under the card.+ The other three values are unchanged and still carry the 4B evidence, which+ is why they are asserted together with this one rather than relaxed with+ it."""from legs import LEGSargs = LEGS["grpo"].argsfor flag, value in (- ("--gpu-memory-utilization", "0.5"),+ ("--gpu-memory-utilization", "0.95"),("--max-seq-length", "1024"),("--num-generations", "2"),("--lora-rank", "16"),@@ -3097,6 +3290,12 @@ def test_an_alternate_spelling_of_the_step_count_keeps_the_reference_band(tmp_pasource, _, name = inner.rpartition(".")if source == "steps.stepcount.outputs":return validated.get(name, "")+ # A dispatch-only or event-only expression. On the pull_request event+ # this test models, `inputs` is null and the schedule branch is false,+ # so GitHub substitutes the empty string -- which is what the leg-list+ # override reads as, and is the shape that must keep --all-kernels.+ if inner.startswith("inputs.") or inner.startswith("github."):+ return ""raise AssertionError(f"the build step takes {key} from {expr}. The step count it builds with, "f"and the reference count it compares against, have to be the ones the "@@ -3236,11 +3435,33 @@ def test_the_workflow_is_never_preempted_by_the_capacity_sweeper():# ----------------------------------------------------------------- report+_TWO_LEGS = [+ {"label": "control", "passed": True, "steps": []},+ {"label": "canary", "passed": True, "steps": []},+]+_A_FAILING_LEG = [+ {"label": "control", "passed": False, "failures": ["x"], "steps": []},+ {"label": "canary", "passed": True, "steps": []},+]+# The Studio payload from the merged kernel. Not this reporter's.+_STUDIO = [{"label": "studio-gpu", "passed": False, "failures": ["x"], "assertions": []}]++@pytest.mark.parametrize(- ("verdict", "expected_exit"),- [("pass", 0), ("partial", 0), ("infra", 0), ("fail", 1)],+ ("verdict", "reports", "expected_exit"),+ [+ ("pass", _TWO_LEGS, 0),+ ("partial", [], 0),+ ("infra", [], 0),+ ("fail", _A_FAILING_LEG, 1),+ # Since --with-studio the kernel verdict covers a payload this reporter+ # does not own, so a failing Studio run must not print "Kaggle T4+ # smoke: FAIL" over two green legs. kaggle_studio_ci/report.py turns+ # that red, in the same job, from the same evidence directory.+ ("fail", _TWO_LEGS + _STUDIO, 0),+ ],)-def test_only_a_real_assertion_failure_turns_the_job_red(tmp_path, verdict, expected_exit):+def test_only_a_real_assertion_failure_turns_the_job_red(tmp_path, verdict, reports, expected_exit):evidence = tmp_path / "evidence"evidence.mkdir()(evidence / "launch_result.json").write_text(@@ -3250,7 +3471,7 @@ def test_only_a_real_assertion_failure_turns_the_job_red(tmp_path, verdict, expe"reason": "test","slug": "u/s","kernel_state": "COMPLETE",- "reports": [],+ "reports": reports,}))@@ -4258,36 +4479,41 @@ def test_the_frontier_leg_does_not_carry_the_zoo_requirement():)-def test_the_frontier_leg_rides_in_the_one_kernel_rather_than_opening_a_session():- """frontier must never be the reason a second Kaggle session is opened.-- This used to read "the seat that costs nothing", on the basis that a- session bills its wall clock once rather than per card, so the second- kernel's idle T4 carried frontier for free. That is still true about- BILLING and is no longer the binding constraint. The account allows two- concurrent GPU sessions; two kernels took both, and- kaggle-t4-studio-gpu-ci.yml runs on the same account, so the notebook leg- locked Unsloth out for as long as it ran. Everything now packs into one- kernel and frontier queues behind the legs ahead of it.+def test_frontier_is_retired_in_favour_of_the_leg_that_supersedes_it():+ """frontier no longer runs, and this used to assert that it did.- So the property worth pinning is no longer "frontier shares a full kernel"- -- that assertion passes just as happily on a two-session layout -- but- "there is one kernel and frontier is in it".+ The property it pinned was "there is one kernel and frontier is in it",+ which mattered when frontier was the cheapest way to cover a+ latest-everything stack. `vision_fla_compile` covers that stack on a bigger+ model AND asserts the vendored FLA kernels, the Turing attention choice, a+ real vision training run, the merged vision export, a Q8_0 GGUF with its+ mmproj sidecar and inference on the exported file. Everything frontier+ proved is a subset.- It has passed on real hardware in both layouts: transformers 5.15.0 /- trl 1.9.2 paired, and transformers 5.15.1 / trl 1.10.0 on run 32607621452,- ten steps, canary emitted, two fresh processes agreeing bitwise.+ Rewritten rather than deleted, because the thing worth guarding now is that+ the retirement was DELIBERATE: a leg that quietly falls out of KERNELS with+ no entry anywhere is indistinguishable from one dropped by a bad merge."""- sys.path.insert(0, str(CI_DIR))- import legs+ from legs import KERNELS, UNWIRED- wired = [kernel for kernel in legs.KERNELS if "frontier" in kernel]- assert len(wired) == 1, f"frontier appears in {len(wired)} kernels, expected 1"- assert len(legs.KERNELS) == 1, (- f"{len(legs.KERNELS)} kernels means {len(legs.KERNELS)} concurrent Kaggle "- "sessions, and the account only has two. Unsloth needs one of them."+ assert "frontier" not in {name for kernel in KERNELS for name in kernel}, (+ "frontier is back in the wired set; if that is intended, this test and "+ "its UNWIRED note both need rewriting rather than deleting")- assert "frontier" not in legs.UNWIRED+ assert "frontier" in UNWIRED, (+ "frontier is in neither KERNELS nor UNWIRED, which is what a leg "+ "dropped by accident looks like"+ )+ assert (+ "SUPERSEDED" in UNWIRED["frontier"]+ ), "the note must say it was replaced, not that it is broken"+ assert "vision_fla_compile" in UNWIRED["frontier"], "and by what"+ # And the leg it names is really carrying the load.+ assert "vision_fla_compile" in {name for kernel in KERNELS for name in kernel}++ # There is still exactly ONE kernel, which is the half of the old+ # assertion that was never about frontier.+ assert len(KERNELS) == 1, KERNELSdef test_the_pinned_kaggle_client_carries_the_calls_this_workflow_makes():diff --git a/tests/kaggle/test_venv_interpreter_and_overlay_path.py b/tests/kaggle/test_venv_interpreter_and_overlay_path.pynew file mode 100644index 00000000000..0500f4eda2f--- /dev/null+++ b/tests/kaggle/test_venv_interpreter_and_overlay_path.py@@ -0,0 +1,197 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""Two defects that presented as one, and neither names this driver in its+traceback.++Measured over six sessions from ONE commit, deterministic per arm:++| arm | Default leg python | torch | datasets | verdict |+|---|---|---|---|---|+| A (3 runs) | 3.12.13 | 2.10.0+cu128 | 4.3.0 | pass 3/3 |+| B (3 runs) | **3.13.13** | 2.12.1+cu130 | 5.0.1 | **fail 3/3** |++Every other leg in both arms stayed on 3.12.13.++**Defect one: the venv interpreter was never pinned.** uv's default+python-preference is `managed`, so once any managed CPython exists under+``~/.local/share/uv/python`` a bare ``uv venv`` builds on that instead of the+Kaggle image's interpreter. ``--system-site-packages`` is still accepted and+still inherits nothing, because a 3.13 venv cannot see a 3.12 site-packages. The+leg then resolves torch, datasets and pyarrow from PyPI -- a stack no Kaggle+user has -- and pays minutes for the privilege, with nothing red.++**Defect two: the overlay directory was hostile to dill**, which is what turned+the wrong interpreter into a crash. ``dill._dill._is_builtin_module`` pickles a+module by REFERENCE only if its ``__file__`` starts with a sys prefix, ends with+an extension suffix, or contains the literal string ``site-packages``. A plain+``pip install --target /tmp/t4ci_venvs/overlay_X`` satisfies none of the three,+so everything in the overlay is pickled BY VALUE. ``Dataset.from_dict``+fingerprints through dill, ``datasets/utils/_dill.py:_save_arrowTable`` saves+``create_arrowTable``, dill walks its globals into the pyarrow module and dies+on pyarrow's Cython ``MonthDayNano``, whose ``__module__`` is ``builtins``:++ PicklingError: Can't pickle <class 'MonthDayNano'>:+ it's not found as builtins.MonthDayNano++Reproduced on CPU in seconds against a byte-identical package tree with the+DIRECTORY NAME as the only variable: the plain ``--target`` directory raised,+the copy named ``site-packages`` returned a fingerprint.++The rules below are asserted through the GENERATED driver source and, for the+dill half, by executing the real path expression against the real dill+predicate. A rule written against the template would not have caught either.+"""++from __future__ import annotations++import pathlib+import re+import sys+import types++import pytest++ROOT = pathlib.Path(__file__).resolve().parents[2]+SMOKE_DIR = ROOT / "tests" / "kaggle" / "t4_smoke"+sys.path.insert(0, str(ROOT / ".github" / "scripts"))++from kaggle_t4_ci import build_kernel, legs # noqa: E402+++def _driver_source() -> str:+ """The driver as the kernel will actually run it, overlays included."""+ names = ("default", "canary")+ payloads = {+ f"t4_{legs.LEGS[n].name}.ipynb": build_kernel.build_payload_notebook(+ SMOKE_DIR, legs.LEGS[n], unsloth_ref = "main", zoo_ref = "main"+ )+ for n in names+ }+ driver = build_kernel.build_driver(+ payloads,+ per_run_timeout = 3600,+ vram_source = {f"t4_{legs.LEGS[n].name}.ipynb": legs.LEGS[n] for n in names},+ overlays = {f"t4_{legs.LEGS[n].name}.ipynb": tuple(legs.LEGS[n].overlay) for n in names},+ )+ return "".join("".join(c["source"]) for c in driver["cells"])+++# --------------------------------------------------------------------------+# Defect one: the interpreter+# --------------------------------------------------------------------------+++def test_the_venv_is_built_on_the_drivers_own_interpreter():+ """`--system-site-packages` on a different python version is a silent+ no-op, so the flag alone is not the guard. The interpreter is."""+ source = _driver_source()+ body = source.split("def _make_venv(")[1].split("def run_one(")[0]+ cmd = re.search(r"venv_cmd = \[([^\]]*)\]", body)+ assert cmd, "the uv venv command is no longer a literal list; re-read this rule"+ assert '"--python"' in cmd.group(1) and "sys.executable" in cmd.group(1), (+ "uv venv runs without --python, so uv's `managed` python-preference "+ "decides the version. That is how the Default leg came up on 3.13 "+ "against a 3.12 image and inherited no system site-packages at all: "+ f"{cmd.group(1)}"+ )+++def test_the_system_site_flag_is_still_passed():+ """The pin does not replace it. Both are needed: the right interpreter with+ no inheritance resolves the whole stack from PyPI just as expensively."""+ body = _driver_source().split("def _make_venv(")[1].split("def run_one(")[0]+ assert '"--system-site-packages"' in body+++def test_the_resulting_python_version_is_REPORTED():+ """Three sessions went red before anyone read a python version out of a+ leg. A venv on the wrong interpreter is invisible without this."""+ body = _driver_source().split("def _make_venv(")[1].split("def run_one(")[0]+ sentinel = body.split("_VENV ")[1]+ assert '"python"' in sentinel, (+ "the _VENV record does not carry the venv's python version, so the "+ "next wrong-interpreter run is again only visible by inference"+ )+ assert (+ "sys.version_info" in body+ ), "nothing compares the venv against the driver's own interpreter"+++# --------------------------------------------------------------------------+# Defect two: the overlay directory, driven against the real dill predicate+# --------------------------------------------------------------------------+++def _overlay_dir_from_generated_source() -> pathlib.Path:+ """Evaluate the driver's OWN `_ov_dir` expression.++ Not a hand-written copy of the path: the rule below is only worth having if+ it fails when the generated expression changes.+ """+ source = _driver_source()+ line = [ln.strip() for ln in source.splitlines() if ln.strip().startswith("_ov_dir = ")]+ assert len(line) == 1, f"expected one _ov_dir assignment, found {len(line)}"+ scope = {+ "VENV_ROOT": pathlib.Path("/tmp/t4ci_venvs"),+ "pathlib": pathlib,+ "name": "t4_Default.ipynb",+ }+ exec(line[0], scope) # noqa: S102 - the driver's own line, by design+ return scope["_ov_dir"]+++def test_the_overlay_lands_where_dill_will_pickle_by_reference():+ """The real dill predicate, against the real generated path.++ `_is_builtin_module` needs only `__file__`, so a module object standing in+ for pyarrow is enough to execute the decision that crashed the leg.+ """+ dill_dill = pytest.importorskip("dill._dill")+ ov = _overlay_dir_from_generated_source()++ stand_in = types.ModuleType("pyarrow")+ stand_in.__file__ = str(ov / "pyarrow" / "__init__.py")+ assert dill_dill._is_builtin_module(stand_in), (+ f"a package installed at {ov} is pickled BY VALUE by dill, so any leg "+ "whose overlay carries pyarrow dies in Dataset.from_dict with "+ "PicklingError: Can't pickle <class 'MonthDayNano'>"+ )+++def test_the_mutation_this_rule_exists_for_is_actually_caught():+ """The negative control. Without it the rule above could be passing because+ dill answers True for everything, which it does not."""+ dill_dill = pytest.importorskip("dill._dill")+ before = types.ModuleType("pyarrow")+ # The path the driver used until this was found: a --target directory that+ # is neither under a sys prefix nor named site-packages.+ before.__file__ = "/tmp/t4ci_venvs/overlay_t4_Default/pyarrow/__init__.py"+ assert not dill_dill._is_builtin_module(before), (+ "dill no longer distinguishes the two paths, so the fix above may have "+ "become unnecessary -- re-measure before simplifying it away"+ )+++def test_the_path_is_not_merely_a_string_containing_site_packages():+ """`site-packages` has to be the directory packages are installed INTO, not+ a decoration further up the path, or pip lays the tree somewhere dill still+ reads by value."""+ ov = _overlay_dir_from_generated_source()+ assert ov.name == "site-packages", (+ f"the overlay --target is {ov}, whose last component is {ov.name!r}; "+ "dill's check is on the installed module's own __file__"+ )+++def test_the_overlay_target_flag_uses_that_directory():+ """The path can be right and unused. This is the link between them."""+ source = _driver_source()+ install = source.split('"--target", str(_ov_dir)')+ assert len(install) == 2, (+ "the overlay install no longer targets _ov_dir, so the directory the "+ "rules above check is not where packages land"+ )+ assert (+ 'env["PYTHONPATH"] = str(_ov_dir)' in source+ ), "the overlay is installed somewhere the payload never reads"diff --git a/tests/kaggle/test_vision_run.py b/tests/kaggle/test_vision_run.pynew file mode 100644index 00000000000..1ddd037635a--- /dev/null+++ b/tests/kaggle/test_vision_run.py@@ -0,0 +1,344 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.++"""The vision run, and the one way it goes green while testing nothing.++A "vision run" that never puts an image on the GPU is a text run in a costume.+It trains, its loss falls, its adapter updates, and every assertion a text leg+makes passes. TRL will produce exactly that state if+`remove_unused_columns=False` is dropped, because the image column is removed+before the collator ever sees it.++So the rules are read off a REAL collated batch, and the guards below are+calibrated to catch the costume rather than the crash.+"""++from __future__ import annotations++import argparse+import sys+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+PAYLOAD = ROOT / "tests" / "kaggle" / "t4_smoke"+sys.path.insert(0, str(PAYLOAD))++from run_vision_t4 import build_conversations, vision_failures # noqa: E402++SRC = (PAYLOAD / "run_vision_t4.py").read_text(encoding = "utf-8")+++def _args(**over):+ base = dict(+ max_steps = 2,+ require_vision_lora = True,+ export = False,+ )+ base.update(over)+ return argparse.Namespace(**base)+++def _good(**over):+ result = {+ "pixels": {+ "columns": ["input_ids", "labels", "pixel_values"],+ "pixel_keys": ["pixel_values"],+ "pixel_sizes": {"pixel_values": {"numel": 602112, "shape": [1, 3, 448, 448]}},+ },+ "vision_lora": {"vision_module_count": 48, "language_modules": 112},+ "metrics": [{"step": 1, "loss": 3.0}, {"step": 2, "loss": 2.0}],+ "adapter_update": {"before": 0.0, "after": 91.2, "tensors": 864, "changed": True},+ "generated": "\\frac{1}{2}",+ }+ result.update(over)+ return result+++def test_the_measured_shape_passes():+ assert vision_failures(_good(), _args()) == []+++def test_a_batch_with_no_pixels_is_the_headline_failure():+ """The costume. Everything else about this run is healthy."""+ broken = vision_failures(+ _good(pixels = {"columns": ["input_ids", "labels"], "pixel_keys": [], "pixel_sizes": {}}),+ _args(),+ )+ assert broken and "no pixel tensor at all" in broken[0]+++def test_an_empty_pixel_tensor_is_a_failure():+ broken = vision_failures(+ _good(pixels = {"pixel_sizes": {"pixel_values": {"numel": 0, "shape": [0]}}}),+ _args(),+ )+ assert broken and "pixel tensors are empty" in broken[0]+++def test_a_lora_that_never_reached_the_vision_tower_is_a_failure():+ """`finetune_vision_layers=True` is a request, not a result."""+ broken = vision_failures(+ _good(vision_lora = {"vision_module_count": 0, "language_modules": 112}), _args()+ )+ assert broken and "no LoRA module landed" in broken[0]+++def test_the_vision_lora_rule_can_be_turned_off_for_a_language_only_run():+ assert (+ vision_failures(+ _good(vision_lora = {"vision_module_count": 0}), _args(require_vision_lora = False)+ )+ == []+ )+++def test_an_adapter_that_did_not_move_is_a_failure():+ """Starts at exactly zero by construction, so any movement is a real+ optimizer step rather than a tolerance question."""+ broken = vision_failures(+ _good(adapter_update = {"before": 0.0, "after": 0.0, "tensors": 864, "changed": False}),+ _args(),+ )+ assert broken and "did not move" in broken[0]+++def test_empty_or_missing_generation_is_a_failure():+ assert vision_failures(_good(generated = ""), _args())+ assert vision_failures(_good(generated = None), _args())+++def test_a_short_step_count_is_a_failure():+ broken = vision_failures(_good(metrics = [{"step": 1, "loss": 3.0}]), _args())+ assert broken and "expected 2 logged steps" in broken[0]+++def test_a_non_finite_loss_is_a_failure():+ broken = vision_failures(+ _good(metrics = [{"step": 1, "loss": float("nan")}, {"step": 2, "loss": 2.0}]), _args()+ )+ assert broken and "non-finite" in broken[0]+++def test_the_export_rule_only_fires_when_the_export_was_requested():+ assert vision_failures(_good(), _args(export = False)) == []+ broken = vision_failures(_good(), _args(export = True))+ assert broken and "failed" in broken[0]+ assert (+ vision_failures(+ _good(export = {"ok": True, "files": [{"name": "model.safetensors", "mb": 1200.0}]}),+ _args(export = True),+ )+ == []+ )+++def test_an_export_that_reported_ok_and_wrote_nothing_is_a_failure():+ """The gpt-oss lesson: an export can succeed and leave no file anywhere."""+ broken = vision_failures(+ _good(export = {"ok": True, "files": [], "dir": "/tmp/x"}), _args(export = True)+ )+ assert broken and "wrote nothing" in broken[0]+++def test_the_conversation_shape_matches_the_notebook():+ """Built from the notebook's own convert_to_conversation. A different shape+ trains something no notebook produces, which tests the leg."""+ rows = build_conversations([{"image": "IMG", "text": "x^2"}])+ assert len(rows) == 1+ messages = rows[0]["messages"]+ assert messages[0]["role"] == "user"+ kinds = [part["type"] for part in messages[0]["content"]]+ assert "image" in kinds and "text" in kinds+ assert messages[1]["role"] == "assistant"+ assert messages[1]["content"][0]["text"] == "x^2"+++def test_the_four_settings_vision_training_needs_are_all_present():+ """Dropping `remove_unused_columns = False` is the exact edit that turns+ this into a text run: TRL removes the image column before the collator sees+ it, and nothing raises."""+ assert "remove_unused_columns = False" in SRC+ assert 'dataset_text_field = ""' in SRC+ assert 'dataset_kwargs = {"skip_prepare_dataset": True}' in SRC+ assert "UnslothVisionDataCollator(model, tokenizer)" in SRC+++def test_the_pixel_evidence_is_read_before_training():+ """After `trainer.train()` the dataloader has been consumed, and a+ re-created one is not necessarily the object the trainer used."""+ pixels_at = SRC.index('result["pixels"] = pixel_evidence(trainer)')+ train_at = SRC.index("stats = trainer.train()")+ assert pixels_at < train_at+++def test_the_export_does_not_land_in_the_artifact_volume():+ """/kaggle/working is 21GB and a merged 2B is a meaningful fraction of it."""+ assert 'tempfile.mkdtemp(prefix = "vision_export_")' in SRC+ # Scoped to the export BLOCK. A whole-file search matches args.outdir in+ # main(), where it is correct, and the assertion would fail for a reason+ # that has nothing to do with the export.+ # Anchored on the mkdtemp rather than on `if args.export:`, because that+ # string appears FIRST in vision_failures() and the naive split lands in+ # the wrong function -- which is how this assertion failed the first time.+ block = SRC.split("export_dir = tempfile.mkdtemp", 1)[1].split('result["export"] = record', 1)[+ 0+ ]+ assert "args.outdir" not in block, "the merged model is written into the artifact volume"+ assert "model.save_pretrained_merged(export_dir, tokenizer)" in block+++def test_the_train_dataset_is_a_dataset_and_its_images_stay_pil():+ """Two failures in one, both measured rather than guessed.++ TRL 1.x rejects a plain list, which is what the notebook passes:++ TypeError: `train_dataset` must be a `Dataset` or `IterableDataset`,+ got `list`++ And the obvious fix corrupts the data. `Dataset.from_list` Arrow-encodes a+ nested PIL object into a `{bytes, path}` DICT on the way back out, so the+ collator receives something that is not an image and nothing says so.++ `with_transform` applies at access time, keeps the column's Image feature,+ and still satisfies TRL's type check.+ """+ from datasets import Dataset as HFDataset+ from PIL import Image++ from run_vision_t4 import conversation_dataset++ base = HFDataset.from_dict({"image": [Image.new("RGB", (8, 8))], "text": ["x^2"]})+ built = conversation_dataset(base)+ assert isinstance(built, HFDataset), "TRL 1.x rejects anything else"+ image = built[0]["messages"][0]["content"][1]["image"]+ assert isinstance(image, Image.Image), (+ f"the image came back as {type(image).__name__}, which is the silent "+ f"Arrow corruption Dataset.from_list produces"+ )+++def test_from_list_would_have_corrupted_the_images():+ """The negative control. Without it, the test above passes for a+ `with_transform` that happens to work and says nothing about why the+ obvious alternative was rejected."""+ from datasets import Dataset as HFDataset+ from PIL import Image++ from run_vision_t4 import build_conversations++ rows = build_conversations([{"image": Image.new("RGB", (8, 8)), "text": "x^2"}])+ naive = HFDataset.from_list(rows)+ image = naive[0]["messages"][0]["content"][1]["image"]+ assert not isinstance(image, Image.Image), (+ "from_list now preserves PIL, so the with_transform indirection may no "+ "longer be needed; re-check before simplifying"+ )+++def test_a_marker_that_matched_nothing_is_refused_rather_than_answered_no():+ """Measured on `unsloth-probe-vision-train-r2-8ed253`, and it named the+ wrong defect. PEFT calls these parameters `lora_B`, with a capital B; the+ marker was matched against the raw name, so it matched none of the 864 of+ them and summed to zero both before AND after. The run had trained+ perfectly well -- loss 1.13 -> 0.56, a merged 4.3 GB export -- and the+ report said the optimizer applied nothing.++ Zero over zero tensors and zero over 864 tensors are opposite findings and+ read identically, which is why the count is carried.+ """+ broken = vision_failures(+ _good(adapter_update = {"before": 0.0, "after": 0.0, "tensors": 0, "changed": False}),+ _args(),+ )+ assert broken, "a question that was never asked must not pass"+ assert "never asked" in broken[0], broken+ assert "did not move" not in broken[0], (+ "reporting an unmatched marker as an untrained adapter sends the "+ "reader after the wrong bug, which is what happened on hardware"+ )+++def test_adapter_sum_finds_the_capital_b_peft_names():+ """Drives the REAL function, because every rule above is fed a dict written+ by hand and none of them execute the code that produces it. That is exactly+ how the capital-B bug reached hardware."""+ import torch++ from run_vision_t4 import adapter_sum++ class _Stub:+ def named_parameters(self):+ # The names PEFT actually emits, capitals and all.+ yield (+ "base_model.model.visual.blocks.0.attn.qkv.lora_A.default.weight",+ torch.ones(2, 2),+ )+ yield (+ "base_model.model.visual.blocks.0.attn.qkv.lora_B.default.weight",+ torch.full((2, 2), 3.0),+ )+ yield (+ "base_model.model.layers.0.self_attn.q_proj.lora_B.default.weight",+ torch.full((2, 2), 1.0),+ )++ got = adapter_sum(_Stub())+ assert got["tensors"] == 2, got+ assert got["sum"] == 16.0, got+++def test_the_leg_actually_DRIVES_the_vision_run():+ """The gap this closes was live for two rounds: `run_vision_t4.py` was in+ the leg's `files` and nothing ever executed it, so Vision_FLA_compile+ trained TEXT, asserted kernels, and shipped a payload it never ran.++ A file that is copied and not run is the quietest kind of coverage there+ is: every guard in this module passed, on a leg where the image path was+ dead.+ """+ import sys as _sys++ _sys.path.insert(0, str(ROOT / ".github" / "scripts" / "kaggle_t4_ci"))+ import legs # noqa: E402++ leg = legs.LEGS["vision_fla_compile"]+ assert "--vision-run" in leg.args, "the leg ships the payload but never runs it"+ assert "run_vision_t4.py" in leg.files+ assert (+ "--export-gguf" in leg.args+ ), "the merged vision export is the half the text path cannot exercise"+++def test_the_parent_spawns_the_vision_run_after_the_cycles():+ """Two 4bit models resident at once on a 14.56GB card is how a leg becomes+ an OOM blamed on the thing it was testing. It is also what keeps a vision+ failure from reading as a text-training one."""+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")+ assert '"run_vision_t4.py"' in src+ cycles_at = src.index("runs.append(json.loads(report_file.read_text")+ spawn_at = src.index('"run_vision_t4.py"')+ assert cycles_at < spawn_at+++def test_a_vision_run_that_wrote_no_report_is_a_failure_not_a_silence():+ """ "the vision run did not happen" and "the vision run passed" are opposite+ outcomes, and an absent report must not read as the second."""+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")+ assert '"the vision process wrote no report"' in src+ assert 'failures += report["vision_failures"]' in src+++def test_the_vision_step_count_is_pinned_low_rather_than_inherited():+ """A vision step on a T4 is ~100s (317.9s for three, measured on+ unsloth-probe-vision-train-r3). Inheriting the text side's --max-steps+ would quietly add half an hour to the leg."""+ src = (PAYLOAD / "run_t4_smoke.py").read_text(encoding = "utf-8")+ spawn = src[src.index('"run_vision_t4.py"') :]+ spawn = spawn[: spawn.index("subprocess.run(vision_cmd)")]+ # Whitespace-insensitive: the repo's formatter reflows this list to one+ # argument per line, and a guard matching the unformatted spelling goes red+ # on a reformat rather than on a regression. That has now happened twice in+ # this payload, so it is worth doing by default.+ flat = "".join(spawn.split())+ assert '"--max-steps","3",' in flat+ assert "args.max_steps" not in flat, "the text step count must not reach the vision run"