Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.
Python agentaichatgptdeepseekfine-tuninggemmaimage-generationllamallmllmsopenaipythonqwenreinforcement-learningself-hostedstable-diffusiontext-to-speechttsuiunsloth
Kaggle T4: describe the batched-generation expectation as what it is (#10377)
Daniel Hanc25841f41a3d
diff --git a/tests/kaggle/t4_smoke/run_t4_smoke.py b/tests/kaggle/t4_smoke/run_t4_smoke.pyindex a49cf933443..3a8e9fc27b7 100644--- a/tests/kaggle/t4_smoke/run_t4_smoke.py+++ b/tests/kaggle/t4_smoke/run_t4_smoke.py@@ -840,17 +840,26 @@ def batched_generation(model, tokenizer, prompts, *, max_new_tokens) -> dict:return result-# Models where batched greedy generation is CONFIRMED broken upstream, filed,-# and not this repo's to fix. See unsloth #9708: on both of these, batch sizes-# 2/4/8 fail to reproduce one-at-a-time output with real left padding and+# Models where batched greedy generation is EXPECTED to disagree with+# one-at-a-time output, and why. See unsloth #9708: on both of these, batch+# sizes 2/4/8 fail to reproduce batch-1 greedy text with real left padding and# demonstrably distinct prompt lengths, on both repeats.#-# This is a STRICT expectation, not a mute. A model listed here that starts-# AGREEING fails the leg, with a message saying to delete the entry -- so the-# day #9708 is fixed, CI says so instead of quietly keeping a stale excuse.-# Every other rule in this function stays live for these models: the padding-# side, the distinct lengths and the empty-output check are what make the-# disagreement a real finding rather than an unpadded batch.+# This is NOT a bug awaiting a fix, in unsloth or anywhere else. It is bf16 /+# fp16 rounding that depends on the shape of the batch: a padded batch takes a+# different SDPA kernel from a lone prompt, the prefill projections run at a+# different GEMM shape, and RMSNorm rounds differently on the result. Plain+# transformers with unsloth never imported shows the same thing, every+# transformers release from 4.57.0 to 5.16.1 shows it, and float32 makes every+# batch size agree. The measurements and a standalone repro are on #9708.+#+# So the entry is a STRICT expectation, not a mute. A model listed here that+# starts AGREEING fails the leg, with a message saying to delete the entry:+# agreement in bf16 means a kernel or a stack changed under us, and that is+# worth hearing about rather than quietly keeping a stale excuse. Every other+# rule in this function stays live for these models: the padding side, the+# distinct lengths and the empty-output checks are what make the disagreement+# a shape effect rather than an unpadded batch or a #9848 empty row.KNOWN_BATCHED_GENERATION_BREAKAGE = {"unsloth/gemma-4-E2B-it": "unsloth#9708","unsloth/Qwen3.5-2B": "unsloth#9708",@@ -902,9 +911,10 @@ def batched_generation_failures(batch: dict | None, model: str | None = None) -># a mute outlives the bug it was written for.out.append(f"{model} is listed in KNOWN_BATCHED_GENERATION_BREAKAGE for "- f"{known}, and every batch size AGREED. If the upstream fix has "- f"landed, delete the entry; the leg must not carry an excuse for a "- f"bug that is gone"+ f"{known}, and every batch size AGREED. That disagreement is bf16 "+ f"rounding that depends on batch shape, so agreement means a kernel "+ f"or the stack changed; work out what, then delete the entry rather "+ f"than carry a stale expectation")return outdiff --git a/tests/kaggle/test_known_upstream_breakage.py b/tests/kaggle/test_known_upstream_breakage.pyindex 557e052ccbc..65adff29994 100644--- a/tests/kaggle/test_known_upstream_breakage.py+++ b/tests/kaggle/test_known_upstream_breakage.py@@ -1,14 +1,15 @@# SPDX-License-Identifier: AGPL-3.0-only# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.-"""A confirmed upstream bug must not block a leg, and must not outlive itself.+"""An expected disagreement must not block a leg, and must not outlive itself.Batched greedy generation on `unsloth/gemma-4-E2B-it` and `unsloth/Qwen3.5-2B`disagrees at sizes 2, 4 and 8 with real left padding and demonstrably distinct-prompt lengths, on both repeats, on a current stack. It is filed as unsloth-#9708 and it is not this repo's to fix. Leaving it as a hard failure means-those two legs put a red in front of every PR for a bug no reader can act on,-which is how a check gets switched off.+prompt lengths, on both repeats, on a current stack. Per unsloth #9708 that is+bf16 rounding that depends on batch shape, reproduced with plain transformers+and gone in float32, so there is no fix coming from anywhere. Leaving it as a+hard failure means those two legs put a red in front of every PR for a+property no reader can act on, which is how a check gets switched off.So the entry is a STRICT expectation rather than a mute, and the difference isthe whole point of this file. A model listed as broken that starts AGREEING@@ -65,8 +66,9 @@ def test_a_listed_model_does_not_fail_on_the_known_disagreement():def test_a_listed_model_that_starts_AGREEING_fails():- """The strict half, and the reason this is not a mute. The day #9708 is- fixed, CI says so instead of carrying a stale excuse."""+ """The strict half, and the reason this is not a mute. Agreement in bf16+ means a kernel or the stack changed, and CI says so instead of carrying a+ stale expectation."""agreeing = _record(agrees = {"2": True, "4": True, "8": True})broken = batched_generation_failures(agreeing, "unsloth/Qwen3.5-2B")assert broken, "a fixed upstream bug must turn the leg red"