Skip to content

Open-source State-of-the-art TTS model which runs on a CPU 😻

Python

Add an opt-in float16 fallback for compute capability 7.5 GPUs

haaziq-sl67f07be199e9
diff --git a/README.md b/README.md
index 0b9c897..24c3cd0 100644
--- a/README.md
+++ b/README.md
@@ -214,7 +214,9 @@ print(model.last_report) # Generation timings and token IDs.
## Running with vLLM
For GPU inference with vLLM on Linux and an NVIDIA CUDA GPU with compute capability
-8.0 or newer (Ampere or later, for bfloat16), install the optional backend:
+8.0 or newer (Ampere or later, for bfloat16), install the optional backend. GPUs with compute
+capability 7.5 can use a float16 fallback that is supported but not the optimal path
+([vLLM guide](docs/vllm.md#older-gpus-compute-capability-75)).
```bash
pip install "kittenml[vllm]"
diff --git a/docs/vllm.md b/docs/vllm.md
index 334f594..2748932 100644
--- a/docs/vllm.md
+++ b/docs/vllm.md
@@ -2,7 +2,9 @@
The optional `vllm` backend runs KittenTTS 2's language model with vLLM 0.31.0 on an NVIDIA GPU;
audio decoding, voice cloning and text preparation stay the same. It needs Linux and a GPU with
-compute capability 8.0 or newer (Ampere or later); older GPUs are refused before any download.
+compute capability 8.0 or newer (Ampere or later). GPUs with compute capability 7.5 can use a
+float16 fallback that is supported but not the optimal path
+([Older GPUs](#older-gpus-compute-capability-75)); older GPUs are refused before any download.
Install and select it as shown in the [README](../README.md#running-with-vllm). That install gets
CUDA 13.0 builds of vLLM and PyTorch, which need a driver for CUDA 13.0
@@ -25,7 +27,7 @@ uv pip install --torch-backend cu129 "kittenml[vllm]" ./vllm-0.31.0+cu129-cp38-a
| `kv_cache_memory_bytes` | not set | A fixed cache size in bytes, used instead of sizing the cache from `gpu_memory_utilization`. |
| `enforce_eager` | `False` | Turns off CUDA graphs. Keep it off: with it on, 4-bit can be slower than the default bf16. |
| `quantization` | not set | `"w4a16"` for 4-bit weights; see [4-bit weights](#4-bit-weights). |
-| `dtype` | `"bfloat16"` | Precision of the language model. Only `"bfloat16"` is accepted; `"float16"` is refused. |
+| `dtype` | `"bfloat16"` | Precision of the language model. `"float16"` is a fallback for GPUs without bfloat16; see [Older GPUs](#older-gpus-compute-capability-75). |
Pass these as `vllm_options` when constructing the model, for example
`vllm_options={"max_model_len": 2048}`; unknown option names are refused before any download. Select
@@ -56,7 +58,8 @@ gives the saved memory to its cache; to lower total use, also set `kv_cache_memo
`gpu_memory_utilization`. It works with `weights="packed"` (the default) or `"full"`, and also
with `weights="emb4"`. That one is lossy: its embedding is stored in 4 bits (at export, perplexity
13.60 against 13.50), and it only makes the download smaller, not GPU memory, because the
-embedding is expanded to bf16 on the GPU either way. KittenTTS checks that vLLM runs the 4-bit layers
+embedding is expanded to bf16 on the GPU either way. `"emb4"` is not available with the float16
+fallback below. KittenTTS checks that vLLM runs the 4-bit layers
with the kernel they were verified with, and refuses settings that would change it.
The biggest gain is with one request at a time: the language model ran about 1.4x faster on an RTX
@@ -71,6 +74,41 @@ both, five sentences one at a time:
| Median time per sentence | 2.65 s | 1.73 s |
| Language-model speed | 54.5 tokens/s | 97.2 tokens/s |
+## Older GPUs (compute capability 7.5)
+
+GPUs without bfloat16, such as the GeForce RTX 20 and GTX 16 series, can run the language model in
+float16 instead, with or without 4-bit weights:
+
+```python
+m = KittenTTS("KittenML/kitten-tts-2", backend="vllm",
+ vllm_options={"dtype": "float16", "quantization": "w4a16"})
+```
+
+**This is a fallback: it is supported, but it is not the optimal path.** Float16 is used only when
+you ask for it; on GPUs with compute capability 8.0 or newer, keep the default (bfloat16). Only an
+RTX 2070 (8 GB) was measured.
+
+- **Precision change.** The model is published in bfloat16 and overflows in plain float16, so
+ KittenTTS scales its internal values down by an exact power of two. Rounding follows float16
+ instead of bfloat16. Given the same text and tokens, the next-token probabilities differ from
+ bfloat16's by 0.0085 nats on average (KL divergence; 4-bit weights in bfloat16 differ by
+ 0.0047). Word error rates on 36 requests (8 languages, long text, cloning) were 0.9% in
+ bfloat16 and 0.8-0.9% in float16. No listening test was done.
+- **Not the fastest attention kernel.** vLLM's FlashAttention 2 needs compute capability 8.0, so
+ these GPUs use a general kernel; startup logs `Cannot use FA version 2 ...`, which is expected.
+ bfloat16 cannot run on these GPUs, so there is no same-card speed comparison.
+- **More memory.** Float16 keeps a separate copy of the output layer: 618 MiB more than bfloat16.
+
+| RTX 2070 (8 GB), one request at a time | float16 + 4-bit, default settings | float16, `kv_cache_memory_bytes` 1 GiB |
+|---|---:|---:|
+| Language-model speed | 108 tokens/s | 74 tokens/s |
+| Synthesis time per second of audio | 0.30 s | 0.41 s |
+| First audio when streaming (median) | 1.33 s | 1.80 s |
+| Peak GPU memory, whole card | 6,574 MiB | 8,024 MiB |
+| Startup with warm caches | 55 s | 46 s |
+
+Measured with `generate()` on the same 36 texts (29 of them for streaming), one request at a time.
+
## Benchmarks
One RTX 3060 (12 GB), driver 570.195.03 (CUDA 12.8), Ubuntu 24.04, Python 3.12, nothing else on the
diff --git a/kittenml/get_model.py b/kittenml/get_model.py
index 284de9b..840de8e 100644
--- a/kittenml/get_model.py
+++ b/kittenml/get_model.py
@@ -72,7 +72,7 @@ class KittenTTS(KittenTTSOnnx):
from .kittentts2 import KittenTTS2
if backend == "vllm":
from .kittentts2.vllm.backend import check_options, validate_environment
- validate_environment(device)
+ validate_environment(device, dict(vllm_options or {}).get("dtype"))
# Before the download: an unsupported GPU or a misspelled option
# should not cost gigabytes.
check_options(vllm_options, weights or config.get("default_weights", "packed"))
diff --git a/kittenml/kittentts2/vllm/backend.py b/kittenml/kittentts2/vllm/backend.py
index 024c1ea..2a4d185 100644
--- a/kittenml/kittentts2/vllm/backend.py
+++ b/kittenml/kittentts2/vllm/backend.py
@@ -37,7 +37,8 @@ def _engine_environment(forced=None):
os.environ[key] = value
-def validate_environment(device=None):
+def validate_environment(device=None, dtype=None):
+ """Check the platform and GPU; `dtype` is vllm_options['dtype'] as given."""
if sys.platform != "linux":
raise RuntimeError("the vLLM backend requires Linux and an NVIDIA CUDA GPU")
if importlib.util.find_spec("vllm") is None:
@@ -48,9 +49,12 @@ def validate_environment(device=None):
if selected.index not in (None, 0):
raise ValueError("select the GPU with CUDA_VISIBLE_DEVICES and use device='cuda'")
major, minor = torch.cuda.get_device_capability()
- if major < 8:
+ if (major, minor) < (7, 5):
+ raise RuntimeError(f"this GPU (SM{major}{minor}) is not supported by the vLLM backend, "
+ "which needs compute capability 7.5 or newer")
+ if major < 8 and dtype != "float16":
raise RuntimeError(f"this GPU (SM{major}{minor}) has no bfloat16 support, which "
- f"KittenTTS's language model needs: {_NO_FLOAT16}")
+ f"KittenTTS's language model runs in by default. {_FLOAT16_OPT_IN}")
# Engine settings KittenTTS chooses for vLLM, and their defaults.
@@ -59,18 +63,17 @@ _ENGINE_DEFAULTS = dict(gpu_memory_utilization=0.5, max_model_len=4096,
# Options KittenTTS interprets itself. None keeps the default behaviour.
_KITTEN_OPTIONS = ("quantization", "dtype", "kv_cache_memory_bytes")
_QUANTIZATIONS = (None, "w4a16")
-_DTYPES = {None: None, "bfloat16": torch.bfloat16}
+_DTYPES = {None: None, "bfloat16": torch.bfloat16, "float16": torch.float16}
# W4A16 runs on vLLM's Marlin kernel, the one whose exact output was verified.
# vLLM tries Machete first on Hopper (SM90) GPUs, so it is disabled for the
# engine; the loader then checks that every quantized layer uses Marlin.
_W4A16_KERNEL = "MarlinLinearKernel"
_W4A16_DISABLED_KERNELS = ("MacheteLinearKernel",)
-# Measured with the current checkpoint, with and without W4A16: in float16 the
-# language model's residual stream passes float16's maximum (65,504) at layer 10
-# and overflows, after which it emits no audio tokens.
-_NO_FLOAT16 = ("this model's residual stream exceeds the float16 range (measured overflow "
- "from layer 10); GPUs without bfloat16 (compute capability < 8.0) are not "
- "supported by the vLLM backend")
+# GPUs without bfloat16 (compute capability 7.5) can opt in to float16, a
+# rescaled fallback with a precision change (fp16.py).
+_FLOAT16_OPT_IN = ("To run on it anyway, opt in to the float16 fallback with "
+ "vllm_options={'dtype': 'float16'}: it is supported but not the optimal "
+ "path (a precision change from bfloat16; see the vLLM backend docs)")
def resolve_options(options):
@@ -83,10 +86,8 @@ def resolve_options(options):
if kitten["quantization"] not in _QUANTIZATIONS:
raise ValueError(f"vllm_options['quantization'] must be None or 'w4a16', "
f"not {kitten['quantization']!r}")
- if kitten["dtype"] == "float16":
- raise ValueError(f"vllm_options['dtype']='float16' is not supported: {_NO_FLOAT16}")
if kitten["dtype"] not in _DTYPES:
- raise ValueError(f"vllm_options['dtype'] must be None or 'bfloat16', "
+ raise ValueError(f"vllm_options['dtype'] must be None, 'bfloat16' or 'float16', "
f"not {kitten['dtype']!r}")
memory = kitten["kv_cache_memory_bytes"]
if memory is not None and (isinstance(memory, bool) or not isinstance(memory, int)
@@ -104,6 +105,10 @@ def check_options(options, weights):
defaults, kitten = resolve_options(options)
if kitten["quantization"] == "w4a16":
check_w4a16_environment()
+ if weights == "emb4" and kitten["dtype"] == "float16":
+ # The float16 rescale was measured on the exact embedding (fp16.py).
+ raise ValueError("vllm_options['dtype']='float16' cannot be combined with "
+ "weights='emb4'; use weights='packed' or 'full'")
return defaults, kitten
@@ -145,9 +150,18 @@ def quantization_args(quantization):
"hf_overrides": {"quantization_config": quantization_config()}}
+def engine_args(quantization, float16):
+ """quantization_args, plus the float16 rescale's model-config changes when asked."""
+ args = quantization_args(quantization)
+ if float16:
+ from .fp16 import hf_overrides
+ args["hf_overrides"] = {**args.get("hf_overrides", {}), **hf_overrides()}
+ return args
+
+
class VLLMBackend:
def __init__(self, owner, options=None):
- validate_environment(owner.device)
+ validate_environment(owner.device, dict(options or {}).get("dtype"))
from transformers import AutoConfig, AutoTokenizer
from ..speaker import load_speaker_head
@@ -157,8 +171,10 @@ class VLLMBackend:
default_dtype = (torch.bfloat16 if owner.config.get("dtype", "bf16") == "bf16"
else torch.float32)
dtype = _DTYPES[kitten["dtype"]] or default_dtype
- if quantization == "w4a16" and dtype is not torch.bfloat16:
- raise ValueError(f"W4A16 needs bfloat16 activations, not {dtype}")
+ # Only an explicit opt-in runs the rescaled float16 model (fp16.py).
+ float16 = dtype is torch.float16
+ if quantization == "w4a16" and dtype not in (torch.bfloat16, torch.float16):
+ raise ValueError(f"W4A16 needs bfloat16 or float16 activations, not {dtype}")
self.max_model_len = defaults["max_model_len"]
self.token_map = owner.token_map
lm_dir = owner.repo_dir / owner.config.get("lm_dir", "lm")
@@ -170,16 +186,28 @@ class VLLMBackend:
# vLLM's own loader reads the full LM; this process only needs the
# embedding and the speaker head, so do not expand the rest here.
state = owner._load_lm_weights(lm_dir, include=_main_process_tensor)
- self.embedding = state["model.embed_tokens.weight"].to(owner.dtype)
- self.spk_proj = load_speaker_head(config.hidden_size, str(lm_dir),
- owner.device, owner.dtype,
- spk_dim=owner.config.get("speaker_embedding_dim", 512),
- state=state)
+ if float16:
+ # Prompt embeddings at the engine's residual scale, built exactly as
+ # the engine's own embedding table; the speaker head runs in float32.
+ from .fp16 import ScaledSpeakerHead, scale_embedding
+ self.embedding = scale_embedding(state["model.embed_tokens.weight"])
+ self.spk_proj = ScaledSpeakerHead(load_speaker_head(
+ config.hidden_size, str(lm_dir), owner.device, torch.float32,
+ spk_dim=owner.config.get("speaker_embedding_dim", 512), state=state))
+ else:
+ self.embedding = state["model.embed_tokens.weight"].to(owner.dtype)
+ self.spk_proj = load_speaker_head(config.hidden_size, str(lm_dir),
+ owner.device, owner.dtype,
+ spk_dim=owner.config.get("speaker_embedding_dim",
+ 512),
+ state=state)
del state
loader_config = {"repo_dir": str(owner.repo_dir), "config": owner.config,
"weights": owner.weights}
if quantization is not None:
loader_config["quantization"] = quantization
+ if float16:
+ loader_config["float16_rescale"] = True
with _engine_environment(forced_environment):
import vllm.envs
from vllm import LLM
@@ -196,7 +224,7 @@ class VLLMBackend:
enable_prompt_embeds=True, enable_prefix_caching=False,
logits_processors=["kittenml.kittentts2.vllm.logits:KittenLogitsProcessor"],
logging_config=LoggingConfig(), disable_log_stats=True,
- **quantization_args(quantization), **defaults)
+ **engine_args(quantization, float16), **defaults)
def prepare_request(self, prompt, speaker_embedding, settings, max_new_tokens, advanced):
from vllm import SamplingParams
diff --git a/kittenml/kittentts2/vllm/fp16.py b/kittenml/kittentts2/vllm/fp16.py
new file mode 100644
index 0000000..95afb87
--- /dev/null
+++ b/kittenml/kittentts2/vllm/fp16.py
@@ -0,0 +1,193 @@
+"""Opt-in float16 mode for GPUs without bfloat16 (compute capability 7.5).
+
+``vllm_options={"dtype": "float16"}`` is a fallback with a precision change:
+the language model was trained and is published in bfloat16, whose range
+float16 (largest value 65,504) does not have. Measured in bfloat16 over a
+multilingual, long-form and cloning corpus, the residual stream (the running
+sum every layer adds to) reaches 69,376 after layer 9 and 3,957,760 at the
+final norm, and the MLP's inner activation (the ``down_proj`` input) reaches
+154,624. Run as is in float16 these overflow and the model emits no speech.
+
+The fix rescales the model exactly instead of changing what it computes:
+
+- Every read of the residual stream goes through an RMSNorm, and
+ ``RMSNorm(c * x)`` with epsilon ``c**2 * eps`` equals ``RMSNorm(x)`` with
+ ``eps``. So the whole stream can be carried at ``2**-9`` of its value: the
+ token embeddings and the speaker projection (the only inputs to the stream)
+ are multiplied by ``2**-9``, so are the outputs of every attention and MLP
+ branch, and the epsilon of the 57 norms that read the stream is multiplied
+ by ``2**-18``. The attention and MLP inputs are normed, so they see the
+ same values as in bfloat16.
+- Each branch's ``2**-9`` is split over two weights, so that the activation
+ inside the branch is scaled too: ``v_proj`` x ``2**-3`` and ``o_proj`` x
+ ``2**-6`` (attention is linear in V), ``up_proj`` x ``2**-5`` and
+ ``down_proj`` x ``2**-4`` (silu(gate) * up is linear in up). Multiplying by
+ a power of two only changes the exponent, and every scaled weight of the
+ published checkpoint is still exactly representable in float16 (checked at
+ load time; the loader refuses otherwise).
+- The output head shares the embedding matrix in the checkpoint. It is
+ untied here and keeps the unscaled values, so the logits are not scaled
+ either (the final norm already removes the stream's scale).
+
+What does change is float16 rounding: activations and weights are stored
+with float16's precision instead of bfloat16's, and the scaled embedding
+(``2**-9`` of values around 0.03) partly falls into float16's subnormal
+range, where 1.35% of its elements are rounded (worst error 4.6e-4 of a
+token's RMS, a quarter of bfloat16's own rounding step). The untied head
+costs one extra copy of the embedding (158,241 x 2,048, 618 MiB).
+
+The scales were chosen from those measurements: ``2**-9`` leaves 8.5x
+headroom below 65,504 at the largest value seen, and the MLP's inner
+activation peaks at 4,832 (13.6x). They are specific to the published
+checkpoint; another checkpoint needs new measurements.
+
+This module does not import vLLM, so it can be tested on any machine.
+"""
+
+import re
+
+import torch
+
+# The residual stream is carried at 2**-RESIDUAL_SHIFT of its bfloat16 value.
+RESIDUAL_SHIFT = 9
+# Weight -> power-of-two shift (multiply by 2**-shift). Each branch's two
+# shifts add up to RESIDUAL_SHIFT, so its output joins the stream at the
+# stream's scale.
+PROJECTION_SHIFTS = {"self_attn.v_proj": 3, "self_attn.o_proj": 6,
+ "mlp.up_proj": 5, "mlp.down_proj": 4}
+# RMSNorm(c x; c^2 eps) == RMSNorm(x; eps).
+EPS_SHIFT = 2 * RESIDUAL_SHIFT
+
+EMBEDDING = "model.embed_tokens.weight"
+LM_HEAD = "lm_head.weight"
+
+# The norms that read the residual stream. q_norm/k_norm read projections of
+# an already normed input and keep their epsilon.
+_RESIDUAL_NORM = re.compile(
+ r"^model\.(?:layers\.\d+\.(?:input_layernorm|post_attention_layernorm)|norm)$")
+# A scaled projection, dense (".weight") or as W4A16 group scales (".weight_scale").
+_SCALED = re.compile(r"^model\.layers\.\d+\.(self_attn\.(?:v|o)_proj|mlp\.(?:up|down)_proj)"
+ r"\.(weight|weight_scale)$")
+# Every projection weight (dense) or W4A16 group scale: converted to float16
+# here, so a lossy conversion is refused rather than left to vLLM's copy.
+_PROJECTION = re.compile(r"^model\.layers\.\d+\.(?:self_attn\.(?:q|k|v|o)_proj|"
+ r"mlp\.(?:gate|up|down)_proj)\.(?:weight|weight_scale)$")
+
+# Written into the engine's model config (hf_overrides). vLLM keys its
+# torch.compile cache on that config, so a float16 engine never reuses graphs
+# compiled with another epsilon, and a change to this plan gets new ones.
+PLAN = (f"residual 2^-{RESIDUAL_SHIFT}; "
+ + ", ".join(f"{name.split('.')[-1]} 2^-{shift}"
+ for name, shift in PROJECTION_SHIFTS.items())
+ + f"; residual-norm eps x 2^-{EPS_SHIFT}; untied unscaled lm_head")
+PLAN_KEY = "kittenml_float16_rescale"
+
+if any(shift != RESIDUAL_SHIFT for shift in (
+ PROJECTION_SHIFTS["self_attn.v_proj"] + PROJECTION_SHIFTS["self_attn.o_proj"],
+ PROJECTION_SHIFTS["mlp.up_proj"] + PROJECTION_SHIFTS["mlp.down_proj"])):
+ raise AssertionError("each branch must be scaled by exactly 2**-RESIDUAL_SHIFT")
+
+
+def hf_overrides():
+ """Model-config changes for the engine: an untied head, and the plan."""
+ return {"tie_word_embeddings": False, PLAN_KEY: PLAN}
+
+
+def shift_of(name):
+ """The power-of-two shift applied to tensor `name` (0 when it is not scaled)."""
+ if name == EMBEDDING:
+ return RESIDUAL_SHIFT
+ match = _SCALED.match(name)
+ return PROJECTION_SHIFTS[match.group(1)] if match else 0
+
+
+def scaled(tensor, shift):
+ """``tensor * 2**-shift`` in float16, rounded once (the scaling itself is exact)."""
+ return (tensor.to(torch.float32) * 2.0 ** -shift).to(torch.float16)
+
+
+def _scaled_and_exact(tensor, shift):
+ """(``scaled(tensor, shift)``, whether that float16 value is exact)."""
+ target = tensor.to(torch.float32) * 2.0 ** -shift
+ result = target.to(torch.float16)
+ # Compare bits so a -0.0 that became +0.0 (or the reverse) is not exact.
+ exact = torch.equal(result.to(torch.float32).view(torch.int32), target.view(torch.int32))
+ return result, exact
+
+
+def is_exact(tensor, shift):
+ """True when ``tensor * 2**-shift`` is exactly representable in float16."""
+ return _scaled_and_exact(tensor, shift)[1]
+
+
+def exact(name, tensor, shift):
+ """`scaled`, refusing a weight float16 cannot hold exactly at this scale."""
+ result, ok = _scaled_and_exact(tensor, shift)
+ if not ok:
+ raise ValueError(f"{name} x 2^-{shift} is not exact in float16; the float16 rescale "
+ "was measured for the published checkpoint and does not fit this one")
+ return result
+
+
+def scale_embedding(weight):
+ """Token embeddings as the float16 engine stores them (also used for prompts)."""
+ return scaled(weight, RESIDUAL_SHIFT)
+
+
+def rescale_weights(weights):
+ """Rescale (name, tensor) pairs, dense or W4A16, for the float16 engine.
+
+ - ``model.embed_tokens.weight`` becomes the scaled embedding, and an
+ unscaled float16 copy is added as ``lm_head.weight`` (the untied head);
+ - every projection weight (dense) or W4A16 group scale becomes float16,
+ with its shift for v/o/up/down, and must be exact;
+ - everything else (norm weights, W4A16 codes and shapes) passes through
+ unchanged, and vLLM converts floating-point tensors to float16 itself.
+ """
+ for name, tensor in weights:
+ if name == LM_HEAD:
+ raise ValueError("the float16 rescale expects a checkpoint whose output head is "
+ "tied to the embedding, not a separate lm_head.weight")
+ if name == EMBEDDING:
+ yield name, scale_embedding(tensor)
+ yield LM_HEAD, tensor.to(torch.float16)
+ elif _PROJECTION.match(name):
+ yield name, exact(name, tensor, shift_of(name))
+ else:
+ yield name, tensor
+
+
+class ScaledSpeakerHead(torch.nn.Module):
+ """The speaker projection, run in float32, its output scaled and rounded to float16."""
+
+ def __init__(self, head):
+ super().__init__()
+ self.head = head.float()
+
+ def forward(self, speaker_embedding):
+ return scaled(self.head(speaker_embedding.float()), RESIDUAL_SHIFT)
+
+
+def residual_norms(model):
+ """(name, module) for every RMSNorm that reads the residual stream."""
+ return [(name, module) for name, module in model.named_modules()
+ if _RESIDUAL_NORM.match(name)]
+
+
+def scale_norm_epsilons(model, layers, eps):
+ """Set the residual norms' epsilon to ``eps * 2**-EPS_SHIFT``; returns how many.
+
+ Runs in the engine after the weights are loaded and before the model is
+ compiled or captured. Setting (not multiplying) keeps a second call harmless.
+ """
+ norms = residual_norms(model)
+ if len(norms) != 2 * layers + 1:
+ raise RuntimeError(f"float16 rescale: expected {2 * layers + 1} residual-stream norms, "
+ f"found {len(norms)}")
+ target = eps * 2.0 ** -EPS_SHIFT
+ for name, module in norms:
+ if module.variance_epsilon not in (eps, target):
+ raise RuntimeError(f"float16 rescale: {name} has epsilon {module.variance_epsilon}, "
+ f"not the model's rms_norm_eps {eps}")
+ module.variance_epsilon = target
+ return len(norms)
diff --git a/kittenml/kittentts2/vllm/loader.py b/kittenml/kittentts2/vllm/loader.py
index 3d0fd06..3e8ecba 100644
--- a/kittenml/kittentts2/vllm/loader.py
+++ b/kittenml/kittentts2/vllm/loader.py
@@ -13,9 +13,13 @@ class KittenModelLoader(BaseModelLoader):
def load_weights(self, model, model_config):
options = self.load_config.model_loader_extra_config
quantization = options.get("quantization")
+ # The opt-in float16 fallback for GPUs without bfloat16 (fp16.py).
+ float16 = options.get("float16_rescale", False)
+ if float16:
+ _check_float16_model(model, model_config)
if quantization == "w4a16":
# vLLM chose each layer's kernel when it built the model; check before loading.
- _check_w4a16_kernels(model, model_config)
+ _check_w4a16_kernels(model, model_config, float16)
# Exact 4-bit weights straight from the trits, one tensor at a time.
from .w4a16 import iter_w4a16
weights = iter_w4a16(options["repo_dir"], options["config"], options["weights"])
@@ -25,10 +29,32 @@ class KittenModelLoader(BaseModelLoader):
if not name.startswith("spk_proj."))
else:
raise ValueError(f"unknown KittenTTS quantization {quantization!r}")
+ if float16:
+ from .fp16 import rescale_weights
+ weights = rescale_weights(weights)
loaded = model.load_weights(weights)
missing = set(dict(model.named_parameters())) - loaded - _kv_cache_scale_names(model)
if missing:
raise ValueError(f"weights missing from the checkpoint: {sorted(missing)[:5]}")
+ if float16:
+ from .fp16 import scale_norm_epsilons
+ config = model_config.hf_config
+ scale_norm_epsilons(model, config.num_hidden_layers, config.rms_norm_eps)
+
+
+def _check_float16_model(model, model_config):
+ """The float16 rescale needs a float16 engine with its own, unscaled output head."""
+ import torch
+
+ refuse = "float16 rescale: refusing to load: "
+ if model_config.dtype != torch.float16:
+ raise RuntimeError(refuse + f"the engine dtype is {model_config.dtype}, not float16")
+ head = getattr(getattr(model, "lm_head", None), "weight", None)
+ embedding = getattr(getattr(getattr(model, "model", None), "embed_tokens", None),
+ "weight", None)
+ if getattr(model_config.hf_config, "tie_word_embeddings", True) or head is None \
+ or head is embedding:
+ raise RuntimeError(refuse + "the output head is tied to the scaled embedding")
def _quantized_layers(model):
@@ -39,7 +65,7 @@ def _quantized_layers(model):
yield name, module, kernel
-def _check_w4a16_kernels(model, model_config):
+def _check_w4a16_kernels(model, model_config, float16=False):
"""Every quantized linear layer must run exact W4A16 on Marlin.
vLLM picks the mixed-precision kernel per GPU (Machete on Hopper, Marlin
@@ -53,8 +79,11 @@ def _check_w4a16_kernels(model, model_config):
from vllm.model_executor.kernels.linear import MarlinLinearKernel
refuse = "W4A16: refusing to load: "
- if model_config.dtype != torch.bfloat16:
- raise RuntimeError(refuse + f"the engine dtype is {model_config.dtype}, not bfloat16")
+ # bfloat16, or float16 in the opt-in float16 mode (its group scales are rescaled).
+ expected = torch.float16 if float16 else torch.bfloat16
+ if model_config.dtype != expected:
+ raise RuntimeError(refuse + f"the engine dtype is {model_config.dtype}, "
+ f"not {str(expected).removeprefix('torch.')}")
chosen, activations = {}, {}
for name, _, kernel in _quantized_layers(model):
chosen.setdefault(type(kernel).__name__, []).append(name)
diff --git a/kittenml/kittentts2/vllm/w4a16.py b/kittenml/kittentts2/vllm/w4a16.py
index 117e9d9..fe81ad2 100644
--- a/kittenml/kittentts2/vllm/w4a16.py
+++ b/kittenml/kittentts2/vllm/w4a16.py
@@ -16,6 +16,10 @@ and the KV cache. ``spk_proj`` runs outside vLLM and is dropped here.
The lossy ``emb4`` packing is refused: W4A16 is offered as an exact
representation of the published weights and is never combined with it.
+In the opt-in float16 mode (fp16.py) the same codes are used; the group scales
+of v/o/up/down are multiplied by that mode's powers of two and stored in
+float16, which must be exact.
+
vLLM fuses q/k/v into ``qkv_proj`` and gate/up into ``gate_up_proj``
(``Qwen3ForCausalLM.packed_modules_mapping``), so the 196 source projections
(7 per layer) become 4 quantized modules per layer: 112 for the 28-layer model.
diff --git a/tests/test_vllm.py b/tests/test_vllm.py
index f0744e2..f116f6e 100644
--- a/tests/test_vllm.py
+++ b/tests/test_vllm.py
@@ -306,15 +306,15 @@ _PROJECTIONS = {"self_attn.q_proj": (256, 256), "self_attn.k_proj": (128, 256),
"mlp.down_proj": (256, 384)}
-def _ternary(rows, cols, generator):
+def _ternary(rows, cols, generator, low=1e-4):
signs = torch.randint(-1, 2, (rows, cols), generator=generator)
- scale = (torch.rand(rows, cols // 128, generator=generator) * 0.5 + 1e-4).to(torch.bfloat16)
+ scale = (torch.rand(rows, cols // 128, generator=generator) * 0.5 + low).to(torch.bfloat16)
scale[0, 0] = 0 # an all-zero group has a zero scale
signs[0, :128] = 0
return signs.to(torch.bfloat16) * scale.repeat_interleave(128, dim=1)
-def _w4a16_checkpoint(tmp_path, layers=2, perturb=None):
+def _w4a16_checkpoint(tmp_path, layers=2, perturb=None, low=1e-4):
"""A small Qwen3-shaped LM written as both full bf16 and packed ternary."""
generator = torch.Generator().manual_seed(0)
state = {"model.embed_tokens.weight": torch.randn(32, 256, generator=generator).to(torch.bfloat16),
@@ -328,7 +328,7 @@ def _w4a16_checkpoint(tmp_path, layers=2, perturb=None):
state[prefix + "input_layernorm.weight"] = torch.ones(256, dtype=torch.bfloat16)
state[prefix + "self_attn.q_norm.weight"] = torch.ones(128, dtype=torch.bfloat16)
for name, shape in _PROJECTIONS.items():
- state[prefix + name + ".weight"] = _ternary(*shape, generator)
+ state[prefix + name + ".weight"] = _ternary(*shape, generator, low)
if perturb is not None:
weight = state[perturb]
weight[1, 5] = weight[1, :128].abs().max() / 2 # no longer ternary in that group
@@ -460,6 +460,8 @@ def test_w4a16_accepts_emb4_and_packs_its_ternary_body_exactly(tmp_path):
assert torch.equal(out[key], value)
check_options({"quantization": "w4a16"}, "emb4")
+ with pytest.raises(ValueError, match="float16"):
+ check_options({"dtype": "float16"}, "emb4")
def test_w4a16_targets_112_fused_modules_for_196_projections():
@@ -510,31 +512,39 @@ def test_vllm_options_validate_quantization_dtype_and_cache_size():
resolve_options(bad)
-def test_float16_is_refused_with_the_measured_reason():
- # float16 was measured to overflow at layer 10 with the current checkpoint
- # (with and without W4A16) and then produce no audio, so it is refused.
+def test_float16_is_an_explicit_option():
+ # float16 is the opt-in fallback for GPUs without bfloat16 (fp16.py); it is
+ # never chosen unless asked for.
from kittenml.kittentts2.vllm.backend import resolve_options
for options in ({"dtype": "float16"}, {"dtype": "float16", "quantization": "w4a16"}):
- with pytest.raises(ValueError, match=r"'float16' is not supported: this model's residual "
- r"stream exceeds the float16 range \(measured overflow "
- r"from layer 10\).*compute capability < 8.0"):
- resolve_options(options)
+ engine, kitten = resolve_options(options)
+ assert kitten["dtype"] == "float16" and "dtype" not in engine
+ assert resolve_options(None)[1]["dtype"] is None
-def test_dtype_follows_hardware_and_gpus_without_bfloat16_are_refused():
+def test_dtype_follows_hardware_and_gpus_without_bfloat16_need_the_float16_opt_in():
from kittenml.kittentts2.vllm.backend import validate_environment
with mock.patch("kittenml.kittentts2.vllm.backend.importlib.util.find_spec", return_value=object()), \
mock.patch("torch.cuda.is_available", return_value=True):
for capability in ((8, 0), (8, 6), (12, 0)):
with mock.patch("torch.cuda.get_device_capability", return_value=capability):
- validate_environment()
- for capability, name in (((7, 5), "SM75"), ((7, 0), "SM70")):
+ for dtype in (None, "bfloat16", "float16"):
+ validate_environment(dtype=dtype)
+ with mock.patch("torch.cuda.get_device_capability", return_value=(7, 5)):
+ for dtype in (None, "bfloat16"):
+ with pytest.raises(RuntimeError, match=r"SM75\) has no bfloat16.*opt in to the "
+ r"float16 fallback with vllm_options="
+ r"\{'dtype': 'float16'\}.*not the optimal"):
+ validate_environment(dtype=dtype)
+ validate_environment(dtype="float16")
+ for capability, name in (((7, 0), "SM70"), ((6, 1), "SM61")):
with mock.patch("torch.cuda.get_device_capability", return_value=capability):
- with pytest.raises(RuntimeError, match=name + ".*no bfloat16.*float16 range"
- ".*layer 10.*< 8.0"):
- validate_environment()
+ for dtype in (None, "float16"):
+ with pytest.raises(RuntimeError, match=name + r"\) is not supported by the "
+ r"vLLM backend.*7.5 or newer"):
+ validate_environment(dtype=dtype)
def test_vllm_options_are_checked_before_model_download(monkeypatch):
@@ -543,7 +553,8 @@ def test_vllm_options_are_checked_before_model_download(monkeypatch):
mock.patch("kittenml.kittentts2.vllm.backend.validate_environment"), \
mock.patch("kittenml.get_model.snapshot_download") as download:
for options, weights, match in [({"quantisation": "w4a16"}, None, "unknown vllm_options"),
- ({"dtype": "float16"}, None, "float16 range")]:
+ ({"dtype": "float16"}, "emb4", "float16.*emb4"),
+ ({"dtype": "fp16"}, None, "'float16'")]:
with pytest.raises(ValueError, match=match):
KittenTTS("example/repo", backend="vllm", weights=weights, vllm_options=options)
monkeypatch.setenv("VLLM_MARLIN_INPUT_DTYPE", "int8")
@@ -559,9 +570,13 @@ def test_gpus_without_bfloat16_are_refused_before_model_download():
mock.patch("torch.cuda.is_available", return_value=True), \
mock.patch("torch.cuda.get_device_capability", return_value=(7, 5)), \
mock.patch("kittenml.get_model.snapshot_download") as download:
- with pytest.raises(RuntimeError, match="SM75.*no bfloat16"):
+ with pytest.raises(RuntimeError, match="SM75.*no bfloat16.*'dtype': 'float16'"):
KittenTTS("example/repo", backend="vllm")
download.assert_not_called()
+ # With the opt-in, the checks pass and the download starts.
+ download.side_effect = RuntimeError("download started")
+ with pytest.raises(RuntimeError, match="download started"):
+ KittenTTS("example/repo", backend="vllm", vllm_options={"dtype": "float16"})
def test_w4a16_refuses_marlin_environment_overrides(monkeypatch, tmp_path):
@@ -653,7 +668,7 @@ def test_engine_receives_w4a16_arguments_and_main_process_loads_only_conditionin
os.environ.pop("VLLM_DISABLED_KERNELS", None)
owner, kwargs = run(None)
assert {"quantization", "hf_overrides", "kv_cache_memory_bytes"}.isdisjoint(kwargs)
- assert "quantization" not in kwargs["model_loader_extra_config"]
+ assert set(kwargs["model_loader_extra_config"]) == {"repo_dir", "config", "weights"}
assert kwargs["dtype"] is torch.bfloat16 and owner.dtype is torch.bfloat16
assert kwargs["disabled_kernels"] is None
owner, kwargs = run({"quantization": "w4a16"})
@@ -663,8 +678,11 @@ def test_engine_receives_w4a16_arguments_and_main_process_loads_only_conditionin
owner, kwargs = run({"quantization": "w4a16", "kv_cache_memory_bytes": 1 << 30,
"dtype": "bfloat16"})
assert kwargs["quantization"] == "compressed-tensors"
+ # bfloat16 W4A16: only the quantization config, the head stays tied.
+ assert set(kwargs["hf_overrides"]) == {"quantization_config"}
assert kwargs["hf_overrides"]["quantization_config"]["ignore"] == ["lm_head"]
assert kwargs["model_loader_extra_config"]["quantization"] == "w4a16"
+ assert "float16_rescale" not in kwargs["model_loader_extra_config"]
assert kwargs["kv_cache_memory_bytes"] == 1 << 30
assert kwargs["dtype"] is torch.bfloat16 and owner.dtype is torch.bfloat16
@@ -857,3 +875,321 @@ def test_w4a16_exactness_checks_compare_bits_not_values(tmp_path):
save_file({"model.layers.0.self_attn.o_proj.weight": weight},
tmp_path / "lm" / "model.safetensors")
assert len(list(iter_w4a16(tmp_path, {}, "full"))) == 3
+
+
+# ── Opt-in float16 (vllm_options={"dtype": "float16"}), fp16.py ─────────────────
+
+def _qwen3(layers=2, tied=True):
+ from transformers import Qwen3Config, Qwen3ForCausalLM
+
+ config = Qwen3Config(vocab_size=32, hidden_size=256, intermediate_size=384,
+ num_hidden_layers=layers, num_attention_heads=2, num_key_value_heads=1,
+ head_dim=128, rms_norm_eps=1e-6, tie_word_embeddings=tied,
+ max_position_embeddings=64)
+ config._attn_implementation = "eager"
+ return Qwen3ForCausalLM(config).double().eval()
+
+
+def _realistic_state(model, generator):
+ """Random weights at the published model's scales: small embeddings (mean(x^2)
+ around 1e-3, so an unscaled epsilon would matter) and a residual stream that grows."""
+ state = {}
+ for name, tensor in model.state_dict().items():
+ if name == "lm_head.weight":
+ continue
+ if name.endswith("norm.weight"):
+ value = torch.rand(tensor.shape, generator=generator, dtype=torch.float64) + 0.5
+ elif name == "model.embed_tokens.weight":
+ value = torch.randn(tensor.shape, generator=generator, dtype=torch.float64) * 0.033
+ else:
+ value = torch.randn(tensor.shape, generator=generator, dtype=torch.float64) * 0.5
+ state[name] = value
+ return state
+
+
+def test_float16_rescale_is_the_same_model_in_float64():
+ # The rescale is exact: with the stream's inputs at 2^-9, every branch at 2^-9 and
+ # the residual norms' epsilon at 2^-18, the logits equal the original model's.
+ # Checked on transformers' Qwen3 in float64, with the plan's own functions.
+ from kittenml.kittentts2.vllm.fp16 import (EMBEDDING, LM_HEAD, RESIDUAL_SHIFT,
+ residual_norms, scale_norm_epsilons, shift_of)
+
+ generator = torch.Generator().manual_seed(3)
+ reference = _qwen3(tied=True)
+ state = _realistic_state(reference, generator)
+ reference.load_state_dict(state, strict=False)
+ reference.tie_weights()
+ rescaled = _qwen3(tied=False)
+ rescaled.load_state_dict({**{name: tensor * 2.0 ** -shift_of(name)
+ for name, tensor in state.items()},
+ LM_HEAD: state[EMBEDDING]}, strict=True)
+ assert rescaled.lm_head.weight.data_ptr() != rescaled.model.embed_tokens.weight.data_ptr()
+ assert scale_norm_epsilons(rescaled, layers=2, eps=1e-6) == 5
+ assert [name for name, _ in residual_norms(rescaled)] == [
+ "model.layers.0.input_layernorm", "model.layers.0.post_attention_layernorm",
+ "model.layers.1.input_layernorm", "model.layers.1.post_attention_layernorm",
+ "model.norm"]
+ assert rescaled.model.layers[0].self_attn.q_norm.variance_epsilon == 1e-6
+
+ ids = torch.tensor([[0, 5, 9, 2, 31, 7]])
+ embeds = state[EMBEDDING][ids].clone()
+ embeds[0, 0] = torch.randn(256, generator=generator, dtype=torch.float64) # spk_proj slot
+ with torch.no_grad():
+ expected = reference(inputs_embeds=embeds).logits
+ actual = rescaled(inputs_embeds=embeds * 2.0 ** -RESIDUAL_SHIFT).logits
+ # transformers' RMSNorm computes in float32, so agreement is to float32 rounding.
+ torch.testing.assert_close(actual, expected, rtol=1e-5, atol=1e-5)
+ # Negative control: without the epsilon scale the first norm (embeddings at
+ # 2^-9, mean(x^2) ~ 4e-9 << 1e-6) is visibly wrong.
+ for _, module in residual_norms(rescaled):
+ module.variance_epsilon = 1e-6
+ wrong = rescaled(inputs_embeds=embeds * 2.0 ** -RESIDUAL_SHIFT).logits
+ assert (wrong - expected).abs().max() > 1e-2
+
+
+def test_float16_plan_shifts_each_branch_by_the_residual_scale():
+ from kittenml.kittentts2.vllm import fp16
+
+ p = "model.layers.3."
+ shifts = {name: fp16.shift_of(p + name + ".weight") for name in _PROJECTIONS}
+ assert shifts == {"self_attn.q_proj": 0, "self_attn.k_proj": 0, "self_attn.v_proj": 3,
+ "self_attn.o_proj": 6, "mlp.gate_proj": 0, "mlp.up_proj": 5,
+ "mlp.down_proj": 4}
+ assert fp16.shift_of(p + "mlp.down_proj.weight_scale") == 4
+ assert fp16.shift_of(p + "mlp.down_proj.weight_packed") == 0
+ assert fp16.shift_of("model.embed_tokens.weight") == fp16.RESIDUAL_SHIFT == 9
+ assert fp16.shift_of(p + "input_layernorm.weight") == 0
+ assert fp16.EPS_SHIFT == 18
+ assert fp16.hf_overrides() == {"tie_word_embeddings": False,
+ "kittenml_float16_rescale": fp16.PLAN}
+ assert fp16.PLAN == ("residual 2^-9; v_proj 2^-3, o_proj 2^-6, up_proj 2^-5, down_proj 2^-4; "
+ "residual-norm eps x 2^-18; untied unscaled lm_head")
+
+
+def test_float16_plan_reaches_the_config_vllm_hashes_for_its_compile_cache():
+ # vLLM's ModelConfig.compute_hash ignores hf_overrides itself but hashes
+ # hf_config.to_json_string() after the overrides are applied with setattr.
+ from transformers import Qwen3Config
+ from kittenml.kittentts2.vllm.fp16 import PLAN, hf_overrides
+
+ config = Qwen3Config(tie_word_embeddings=True)
+ before = config.to_json_string()
+ for key, value in hf_overrides().items():
+ setattr(config, key, value)
+ after = config.to_json_string()
+ assert after != before and PLAN in after and config.tie_word_embeddings is False
+
+
[email protected]("variant", ["packed", "full"])
+def test_float16_rescale_dense_and_w4a16_give_the_same_exact_float16_weights(tmp_path, variant):
+ from kittenml.kittentts2.vllm.fp16 import rescale_weights, scaled, shift_of
+ from kittenml.kittentts2.vllm.w4a16 import bit_equal, is_projection, iter_w4a16
+
+ # Group scales >= 2^-6, as in the published checkpoint: every shifted weight
+ # stays a normal float16 number.
+ state, config = _w4a16_checkpoint(tmp_path, low=2.0 ** -6)
+ weights = load_lm_weights(tmp_path, config, variant)
+ dense = dict(rescale_weights((n, t) for n, t in weights.items()
+ if not n.startswith("spk_proj.")))
+ original = dict(iter_w4a16(tmp_path, config, variant))
+ quantized = dict(rescale_weights(original.items()))
+ unpack = _reference_unpack()
+ embedding = state["model.embed_tokens.weight"]
+ for out in (dense, quantized):
+ assert bit_equal(out["model.embed_tokens.weight"], scaled(embedding, 9))
+ assert bit_equal(out["lm_head.weight"], embedding.to(torch.float16)) # untied, unscaled
+ assert bit_equal(out["model.norm.weight"], state["model.norm.weight"]) # unchanged
+ for name in (n for n in state if is_projection(n)):
+ exact = state[name].to(torch.float32) * 2.0 ** -shift_of(name)
+ assert dense[name].dtype == torch.float16
+ assert torch.equal(dense[name].to(torch.float32), exact), name
+ base = name.removesuffix(".weight")
+ scale = quantized[base + ".weight_scale"]
+ assert scale.dtype == torch.float16
+ assert quantized[base + ".weight_packed"] is original[base + ".weight_packed"]
+ assert torch.equal(scale.to(torch.float32), original[base + ".weight_scale"].float()
+ * 2.0 ** -shift_of(name))
+ signs = unpack(quantized[base + ".weight_packed"], state[name].shape)
+ rebuilt = signs.to(torch.float16) * scale.repeat_interleave(128, dim=1)
+ assert bit_equal(rebuilt, dense[name]), name
+
+
+def test_float16_rescale_refuses_weights_it_cannot_hold_exactly():
+ from kittenml.kittentts2.vllm.fp16 import is_exact, rescale_weights
+
+ weight = torch.full((2, 128), 0.25, dtype=torch.bfloat16)
+ weight[1, 7] = 2.0 ** -20 # x 2^-6 = 2^-26: below float16's smallest subnormal
+ pairs = [("model.layers.0.self_attn.o_proj.weight", weight)]
+ with pytest.raises(ValueError, match=r"o_proj.weight x 2\^-6 is not exact in float16"):
+ dict(rescale_weights(pairs))
+ # The same value is exact without a shift (q_proj) ... and -0.0 must stay -0.0.
+ assert len(dict(rescale_weights([("model.layers.0.self_attn.q_proj.weight", weight)]))) == 1
+ zero = torch.tensor([-0.0], dtype=torch.bfloat16)
+ assert is_exact(zero, 6) and is_exact(torch.tensor([65504.0]), 0)
+ assert not is_exact(torch.tensor([65536.0]), 0) # overflow is not exact
+ with pytest.raises(ValueError, match="tied"):
+ dict(rescale_weights([("lm_head.weight", weight)]))
+
+
+def test_float16_prompt_embeddings_match_the_engine_scale():
+ pytest.importorskip("vllm")
+ from kittenml.kittentts2.vllm.backend import VLLMBackend
+ from kittenml.kittentts2.vllm.fp16 import ScaledSpeakerHead, scale_embedding, scaled
+
+ generator = torch.Generator().manual_seed(5)
+ table = (torch.randn(16, 4, generator=generator) * 0.03).to(torch.bfloat16)
+ head = torch.nn.Linear(4, 4).to(torch.bfloat16)
+ backend = object.__new__(VLLMBackend)
+ backend.max_model_len = 32
+ backend.token_map = TokenMap(5, 10, 1, 2, 3, 4, 0, 15)
+ backend.embedding = scale_embedding(table)
+ backend.spk_proj = ScaledSpeakerHead(head)
+ speaker = torch.randn(1, 4, generator=generator).to(torch.float16)
+ settings = dict(temperature=0.7, top_k=5, top_p=0.8, min_p=0.02)
+ advanced = dict(seed=1, repetition_penalty=1.2, repetition_window=3,
+ token_run_penalty=1.5, token_run_grace=4)
+ request, _ = backend.prepare_request([0, 3, 1], speaker, settings, 8, advanced)
+ embeds = request["prompt_embeds"]
+ assert embeds.dtype == torch.float16
+ assert torch.equal(embeds[1:], scaled(table[[3, 1]], 9))
+ expected = scaled(torch.nn.functional.linear(speaker.float(), head.weight.float(),
+ head.bias.float()), 9).squeeze(0)
+ assert torch.equal(embeds[0], expected)
+
+
+def _norm(eps=1e-6):
+ module = torch.nn.Module()
+ module.variance_epsilon = eps
+ return module
+
+
+def _float16_engine_model(layers=2, tied=False):
+ """Just enough of vLLM's Qwen3ForCausalLM for the loader's float16 checks."""
+ model = torch.nn.Module()
+ model.model = torch.nn.Module()
+ model.model.embed_tokens = torch.nn.Embedding(4, 2)
+ model.model.layers = torch.nn.ModuleList()
+ for _ in range(layers):
+ layer = torch.nn.Module()
+ layer.input_layernorm, layer.post_attention_layernorm = _norm(), _norm()
+ layer.self_attn = torch.nn.Module()
+ layer.self_attn.q_norm, layer.self_attn.k_norm = _norm(), _norm()
+ model.model.layers.append(layer)
+ model.model.norm = _norm()
+ model.lm_head = model.model.embed_tokens if tied else torch.nn.Embedding(4, 2)
+ model.loaded = {}
+
+ def load_weights(weights):
+ model.loaded.update(weights)
+ return set(dict(model.named_parameters()))
+ model.load_weights = load_weights
+ return model
+
+
+def test_vllm_loader_float16_rescales_weights_and_residual_norm_epsilon_only():
+ pytest.importorskip("vllm")
+ from kittenml.kittentts2.vllm.loader import KittenModelLoader
+
+ loader = object.__new__(KittenModelLoader)
+ loader.load_config = SimpleNamespace(model_loader_extra_config={
+ "repo_dir": "unused", "config": {}, "weights": "packed", "float16_rescale": True})
+ hf_config = SimpleNamespace(num_hidden_layers=2, rms_norm_eps=1e-6, tie_word_embeddings=False)
+ fp16_config = SimpleNamespace(dtype=torch.float16, hf_config=hf_config)
+ embedding = torch.full((4, 2), 0.03, dtype=torch.bfloat16)
+ state = {"model.embed_tokens.weight": embedding, "spk_proj.0.bias": torch.zeros(2),
+ "model.layers.0.mlp.up_proj.weight": torch.full((2, 2), 0.5, dtype=torch.bfloat16)}
+ model = _float16_engine_model()
+ with mock.patch("kittenml.kittentts2.vllm.loader.load_lm_weights", return_value=state):
+ loader.load_weights(model, fp16_config)
+ assert set(model.loaded) == {"model.embed_tokens.weight", "lm_head.weight",
+ "model.layers.0.mlp.up_proj.weight"}
+ assert torch.equal(model.loaded["lm_head.weight"], embedding.to(torch.float16))
+ assert model.loaded["model.layers.0.mlp.up_proj.weight"].unique().tolist() == [2.0 ** -6]
+ for layer in model.model.layers:
+ assert layer.input_layernorm.variance_epsilon == 1e-6 * 2.0 ** -18
+ assert layer.post_attention_layernorm.variance_epsilon == 1e-6 * 2.0 ** -18
+ assert layer.self_attn.q_norm.variance_epsilon == 1e-6 # reads a normed input
+ assert layer.self_attn.k_norm.variance_epsilon == 1e-6
+ assert model.model.norm.variance_epsilon == 1e-6 * 2.0 ** -18
+ loader.load_weights(model, fp16_config) # a second load does not scale twice
+ assert model.model.norm.variance_epsilon == 1e-6 * 2.0 ** -18
+ with pytest.raises(RuntimeError, match="engine dtype is torch.bfloat16, not float16"):
+ loader.load_weights(_float16_engine_model(),
+ SimpleNamespace(dtype=torch.bfloat16, hf_config=hf_config))
+ with pytest.raises(RuntimeError, match="tied to the scaled embedding"):
+ loader.load_weights(_float16_engine_model(tied=True), fp16_config)
+ with pytest.raises(RuntimeError, match="expected 5 residual-stream norms, found 7"):
+ loader.load_weights(_float16_engine_model(layers=3), fp16_config)
+
+
+def test_vllm_loader_bfloat16_paths_hand_over_the_checkpoint_tensors_untouched():
+ # Without the float16 opt-in the loader passes the very same tensors and never
+ # touches an epsilon, for the dense and the W4A16 path.
+ pytest.importorskip("vllm")
+ from vllm.model_executor.kernels.linear import MarlinLinearKernel
+ from kittenml.kittentts2.vllm.loader import KittenModelLoader
+
+ bf16 = SimpleNamespace(dtype=torch.bfloat16)
+ for quantization in (None, "w4a16"):
+ loader = object.__new__(KittenModelLoader)
+ extra = {"repo_dir": "unused", "config": {}, "weights": "packed"}
+ if quantization:
+ extra["quantization"] = quantization
+ loader.load_config = SimpleNamespace(model_loader_extra_config=extra)
+ model = _float16_engine_model(tied=True)
+ model.proj = _linear_with_kernel(MarlinLinearKernel)
+ tensors = {"model.embed_tokens.weight": torch.full((4, 2), 0.03, dtype=torch.bfloat16),
+ "model.layers.0.mlp.up_proj.weight": torch.ones(2, 2, dtype=torch.bfloat16)}
+ with mock.patch("kittenml.kittentts2.vllm.loader.load_lm_weights", return_value=tensors), \
+ mock.patch("kittenml.kittentts2.vllm.w4a16.iter_w4a16",
+ return_value=iter(tensors.items())):
+ loader.load_weights(model, bf16)
+ assert set(model.loaded) == set(tensors)
+ assert all(model.loaded[name] is tensors[name] for name in tensors)
+ assert all(module.variance_epsilon == 1e-6 for name, module in model.named_modules()
+ if hasattr(module, "variance_epsilon"))
+
+
+def test_engine_receives_the_float16_rescale_only_when_asked(tmp_path):
+ pytest.importorskip("vllm")
+ from kittenml.kittentts2.vllm.backend import VLLMBackend
+ from kittenml.kittentts2.vllm.fp16 import PLAN, ScaledSpeakerHead, scale_embedding
+
+ seen = []
+ embedding = torch.full((2, 4), 0.03, dtype=torch.bfloat16)
+
+ def run(options):
+ owner = SimpleNamespace(device="cpu", repo_dir=tmp_path, config={}, weights="packed",
+ token_map=None, _load_lm_weights=mock.Mock(return_value={
+ "model.embed_tokens.weight": embedding}))
+ with mock.patch("kittenml.kittentts2.vllm.backend.validate_environment") as validate, \
+ mock.patch("transformers.AutoConfig.from_pretrained", return_value=SimpleNamespace(
+ model_type="qwen3", hidden_size=4)), \
+ mock.patch("transformers.AutoTokenizer.from_pretrained"), \
+ mock.patch("kittenml.kittentts2.speaker.load_speaker_head",
+ return_value=torch.nn.Linear(4, 4)) as head, \
+ mock.patch("vllm.LLM", side_effect=lambda **kwargs: seen.append(kwargs)):
+ backend = VLLMBackend(owner, options)
+ validate.assert_called_once_with("cpu", (options or {}).get("dtype"))
+ return owner, backend, head, seen[-1]
+
+ for options in ({"dtype": "float16"}, {"dtype": "float16", "quantization": "w4a16"}):
+ owner, backend, head, kwargs = run(options)
+ assert kwargs["dtype"] is torch.float16 and owner.dtype is torch.float16
+ assert kwargs["model_loader_extra_config"]["float16_rescale"] is True
+ overrides = kwargs["hf_overrides"]
+ assert overrides["tie_word_embeddings"] is False
+ assert overrides["kittenml_float16_rescale"] == PLAN
+ assert ("quantization_config" in overrides) == ("quantization" in options)
+ assert torch.equal(backend.embedding, scale_embedding(embedding))
+ assert isinstance(backend.spk_proj, ScaledSpeakerHead)
+ assert head.call_args.args[3] is torch.float32 # the speaker head runs in float32
+ for options in (None, {"dtype": "bfloat16"}, {"quantization": "w4a16"}):
+ owner, backend, head, kwargs = run(options)
+ assert kwargs["dtype"] is torch.bfloat16 and owner.dtype is torch.bfloat16
+ assert "float16_rescale" not in kwargs["model_loader_extra_config"]
+ assert "tie_word_embeddings" not in kwargs.get("hf_overrides", {})
+ assert backend.embedding.dtype is torch.bfloat16 and torch.equal(backend.embedding,
+ embedding)
+ assert backend.spk_proj is head.return_value
+ assert head.call_args.args[3] is torch.bfloat16