Skip to content

Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.

Kaggle T4: describe the batched-generation expectation as what it is (#10377)

Daniel Hanc25841f41a3d
diff --git a/tests/kaggle/t4_smoke/run_t4_smoke.py b/tests/kaggle/t4_smoke/run_t4_smoke.py
index a49cf933443..3a8e9fc27b7 100644
--- a/tests/kaggle/t4_smoke/run_t4_smoke.py
+++ b/tests/kaggle/t4_smoke/run_t4_smoke.py
@@ -840,17 +840,26 @@ def batched_generation(model, tokenizer, prompts, *, max_new_tokens) -> dict:
return result
-# Models where batched greedy generation is CONFIRMED broken upstream, filed,
-# and not this repo's to fix. See unsloth #9708: on both of these, batch sizes
-# 2/4/8 fail to reproduce one-at-a-time output with real left padding and
+# Models where batched greedy generation is EXPECTED to disagree with
+# one-at-a-time output, and why. See unsloth #9708: on both of these, batch
+# sizes 2/4/8 fail to reproduce batch-1 greedy text with real left padding and
# demonstrably distinct prompt lengths, on both repeats.
#
-# This is a STRICT expectation, not a mute. A model listed here that starts
-# AGREEING fails the leg, with a message saying to delete the entry -- so the
-# day #9708 is fixed, CI says so instead of quietly keeping a stale excuse.
-# Every other rule in this function stays live for these models: the padding
-# side, the distinct lengths and the empty-output check are what make the
-# disagreement a real finding rather than an unpadded batch.
+# This is NOT a bug awaiting a fix, in unsloth or anywhere else. It is bf16 /
+# fp16 rounding that depends on the shape of the batch: a padded batch takes a
+# different SDPA kernel from a lone prompt, the prefill projections run at a
+# different GEMM shape, and RMSNorm rounds differently on the result. Plain
+# transformers with unsloth never imported shows the same thing, every
+# transformers release from 4.57.0 to 5.16.1 shows it, and float32 makes every
+# batch size agree. The measurements and a standalone repro are on #9708.
+#
+# So the entry is a STRICT expectation, not a mute. A model listed here that
+# starts AGREEING fails the leg, with a message saying to delete the entry:
+# agreement in bf16 means a kernel or a stack changed under us, and that is
+# worth hearing about rather than quietly keeping a stale excuse. Every other
+# rule in this function stays live for these models: the padding side, the
+# distinct lengths and the empty-output checks are what make the disagreement
+# a shape effect rather than an unpadded batch or a #9848 empty row.
KNOWN_BATCHED_GENERATION_BREAKAGE = {
"unsloth/gemma-4-E2B-it": "unsloth#9708",
"unsloth/Qwen3.5-2B": "unsloth#9708",
@@ -902,9 +911,10 @@ def batched_generation_failures(batch: dict | None, model: str | None = None) ->
# a mute outlives the bug it was written for.
out.append(
f"{model} is listed in KNOWN_BATCHED_GENERATION_BREAKAGE for "
- f"{known}, and every batch size AGREED. If the upstream fix has "
- f"landed, delete the entry; the leg must not carry an excuse for a "
- f"bug that is gone"
+ f"{known}, and every batch size AGREED. That disagreement is bf16 "
+ f"rounding that depends on batch shape, so agreement means a kernel "
+ f"or the stack changed; work out what, then delete the entry rather "
+ f"than carry a stale expectation"
)
return out
diff --git a/tests/kaggle/test_known_upstream_breakage.py b/tests/kaggle/test_known_upstream_breakage.py
index 557e052ccbc..65adff29994 100644
--- a/tests/kaggle/test_known_upstream_breakage.py
+++ b/tests/kaggle/test_known_upstream_breakage.py
@@ -1,14 +1,15 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
-"""A confirmed upstream bug must not block a leg, and must not outlive itself.
+"""An expected disagreement must not block a leg, and must not outlive itself.
Batched greedy generation on `unsloth/gemma-4-E2B-it` and `unsloth/Qwen3.5-2B`
disagrees at sizes 2, 4 and 8 with real left padding and demonstrably distinct
-prompt lengths, on both repeats, on a current stack. It is filed as unsloth
-#9708 and it is not this repo's to fix. Leaving it as a hard failure means
-those two legs put a red in front of every PR for a bug no reader can act on,
-which is how a check gets switched off.
+prompt lengths, on both repeats, on a current stack. Per unsloth #9708 that is
+bf16 rounding that depends on batch shape, reproduced with plain transformers
+and gone in float32, so there is no fix coming from anywhere. Leaving it as a
+hard failure means those two legs put a red in front of every PR for a
+property no reader can act on, which is how a check gets switched off.
So the entry is a STRICT expectation rather than a mute, and the difference is
the whole point of this file. A model listed as broken that starts AGREEING
@@ -65,8 +66,9 @@ def test_a_listed_model_does_not_fail_on_the_known_disagreement():
def test_a_listed_model_that_starts_AGREEING_fails():
- """The strict half, and the reason this is not a mute. The day #9708 is
- fixed, CI says so instead of carrying a stale excuse."""
+ """The strict half, and the reason this is not a mute. Agreement in bf16
+ means a kernel or the stack changed, and CI says so instead of carrying a
+ stale expectation."""
agreeing = _record(agrees = {"2": True, "4": True, "8": True})
broken = batched_generation_failures(agreeing, "unsloth/Qwen3.5-2B")
assert broken, "a fixed upstream bug must turn the leg red"