Open-source State-of-the-art TTS model which runs on a CPU π»
Python
Add an opt-in float16 fallback for compute capability 7.5 GPUs
haaziq-sl67f07be199e9
diff --git a/README.md b/README.mdindex 0b9c897..24c3cd0 100644--- a/README.md+++ b/README.md@@ -214,7 +214,9 @@ print(model.last_report) # Generation timings and token IDs.## Running with vLLMFor GPU inference with vLLM on Linux and an NVIDIA CUDA GPU with compute capability-8.0 or newer (Ampere or later, for bfloat16), install the optional backend:+8.0 or newer (Ampere or later, for bfloat16), install the optional backend. GPUs with compute+capability 7.5 can use a float16 fallback that is supported but not the optimal path+([vLLM guide](docs/vllm.md#older-gpus-compute-capability-75)).```bashpip install "kittenml[vllm]"diff --git a/docs/vllm.md b/docs/vllm.mdindex 334f594..2748932 100644--- a/docs/vllm.md+++ b/docs/vllm.md@@ -2,7 +2,9 @@The optional `vllm` backend runs KittenTTS 2's language model with vLLM 0.31.0 on an NVIDIA GPU;audio decoding, voice cloning and text preparation stay the same. It needs Linux and a GPU with-compute capability 8.0 or newer (Ampere or later); older GPUs are refused before any download.+compute capability 8.0 or newer (Ampere or later). GPUs with compute capability 7.5 can use a+float16 fallback that is supported but not the optimal path+([Older GPUs](#older-gpus-compute-capability-75)); older GPUs are refused before any download.Install and select it as shown in the [README](../README.md#running-with-vllm). That install getsCUDA 13.0 builds of vLLM and PyTorch, which need a driver for CUDA 13.0@@ -25,7 +27,7 @@ uv pip install --torch-backend cu129 "kittenml[vllm]" ./vllm-0.31.0+cu129-cp38-a| `kv_cache_memory_bytes` | not set | A fixed cache size in bytes, used instead of sizing the cache from `gpu_memory_utilization`. || `enforce_eager` | `False` | Turns off CUDA graphs. Keep it off: with it on, 4-bit can be slower than the default bf16. || `quantization` | not set | `"w4a16"` for 4-bit weights; see [4-bit weights](#4-bit-weights). |-| `dtype` | `"bfloat16"` | Precision of the language model. Only `"bfloat16"` is accepted; `"float16"` is refused. |+| `dtype` | `"bfloat16"` | Precision of the language model. `"float16"` is a fallback for GPUs without bfloat16; see [Older GPUs](#older-gpus-compute-capability-75). |Pass these as `vllm_options` when constructing the model, for example`vllm_options={"max_model_len": 2048}`; unknown option names are refused before any download. Select@@ -56,7 +58,8 @@ gives the saved memory to its cache; to lower total use, also set `kv_cache_memo`gpu_memory_utilization`. It works with `weights="packed"` (the default) or `"full"`, and alsowith `weights="emb4"`. That one is lossy: its embedding is stored in 4 bits (at export, perplexity13.60 against 13.50), and it only makes the download smaller, not GPU memory, because the-embedding is expanded to bf16 on the GPU either way. KittenTTS checks that vLLM runs the 4-bit layers+embedding is expanded to bf16 on the GPU either way. `"emb4"` is not available with the float16+fallback below. KittenTTS checks that vLLM runs the 4-bit layerswith the kernel they were verified with, and refuses settings that would change it.The biggest gain is with one request at a time: the language model ran about 1.4x faster on an RTX@@ -71,6 +74,41 @@ both, five sentences one at a time:| Median time per sentence | 2.65 s | 1.73 s || Language-model speed | 54.5 tokens/s | 97.2 tokens/s |+## Older GPUs (compute capability 7.5)++GPUs without bfloat16, such as the GeForce RTX 20 and GTX 16 series, can run the language model in+float16 instead, with or without 4-bit weights:++```python+m = KittenTTS("KittenML/kitten-tts-2", backend="vllm",+ vllm_options={"dtype": "float16", "quantization": "w4a16"})+```++**This is a fallback: it is supported, but it is not the optimal path.** Float16 is used only when+you ask for it; on GPUs with compute capability 8.0 or newer, keep the default (bfloat16). Only an+RTX 2070 (8 GB) was measured.++- **Precision change.** The model is published in bfloat16 and overflows in plain float16, so+ KittenTTS scales its internal values down by an exact power of two. Rounding follows float16+ instead of bfloat16. Given the same text and tokens, the next-token probabilities differ from+ bfloat16's by 0.0085 nats on average (KL divergence; 4-bit weights in bfloat16 differ by+ 0.0047). Word error rates on 36 requests (8 languages, long text, cloning) were 0.9% in+ bfloat16 and 0.8-0.9% in float16. No listening test was done.+- **Not the fastest attention kernel.** vLLM's FlashAttention 2 needs compute capability 8.0, so+ these GPUs use a general kernel; startup logs `Cannot use FA version 2 ...`, which is expected.+ bfloat16 cannot run on these GPUs, so there is no same-card speed comparison.+- **More memory.** Float16 keeps a separate copy of the output layer: 618 MiB more than bfloat16.++| RTX 2070 (8 GB), one request at a time | float16 + 4-bit, default settings | float16, `kv_cache_memory_bytes` 1 GiB |+|---|---:|---:|+| Language-model speed | 108 tokens/s | 74 tokens/s |+| Synthesis time per second of audio | 0.30 s | 0.41 s |+| First audio when streaming (median) | 1.33 s | 1.80 s |+| Peak GPU memory, whole card | 6,574 MiB | 8,024 MiB |+| Startup with warm caches | 55 s | 46 s |++Measured with `generate()` on the same 36 texts (29 of them for streaming), one request at a time.+## BenchmarksOne RTX 3060 (12 GB), driver 570.195.03 (CUDA 12.8), Ubuntu 24.04, Python 3.12, nothing else on thediff --git a/kittenml/get_model.py b/kittenml/get_model.pyindex 284de9b..840de8e 100644--- a/kittenml/get_model.py+++ b/kittenml/get_model.py@@ -72,7 +72,7 @@ class KittenTTS(KittenTTSOnnx):from .kittentts2 import KittenTTS2if backend == "vllm":from .kittentts2.vllm.backend import check_options, validate_environment- validate_environment(device)+ validate_environment(device, dict(vllm_options or {}).get("dtype"))# Before the download: an unsupported GPU or a misspelled option# should not cost gigabytes.check_options(vllm_options, weights or config.get("default_weights", "packed"))diff --git a/kittenml/kittentts2/vllm/backend.py b/kittenml/kittentts2/vllm/backend.pyindex 024c1ea..2a4d185 100644--- a/kittenml/kittentts2/vllm/backend.py+++ b/kittenml/kittentts2/vllm/backend.py@@ -37,7 +37,8 @@ def _engine_environment(forced=None):os.environ[key] = value-def validate_environment(device=None):+def validate_environment(device=None, dtype=None):+ """Check the platform and GPU; `dtype` is vllm_options['dtype'] as given."""if sys.platform != "linux":raise RuntimeError("the vLLM backend requires Linux and an NVIDIA CUDA GPU")if importlib.util.find_spec("vllm") is None:@@ -48,9 +49,12 @@ def validate_environment(device=None):if selected.index not in (None, 0):raise ValueError("select the GPU with CUDA_VISIBLE_DEVICES and use device='cuda'")major, minor = torch.cuda.get_device_capability()- if major < 8:+ if (major, minor) < (7, 5):+ raise RuntimeError(f"this GPU (SM{major}{minor}) is not supported by the vLLM backend, "+ "which needs compute capability 7.5 or newer")+ if major < 8 and dtype != "float16":raise RuntimeError(f"this GPU (SM{major}{minor}) has no bfloat16 support, which "- f"KittenTTS's language model needs: {_NO_FLOAT16}")+ f"KittenTTS's language model runs in by default. {_FLOAT16_OPT_IN}")# Engine settings KittenTTS chooses for vLLM, and their defaults.@@ -59,18 +63,17 @@ _ENGINE_DEFAULTS = dict(gpu_memory_utilization=0.5, max_model_len=4096,# Options KittenTTS interprets itself. None keeps the default behaviour._KITTEN_OPTIONS = ("quantization", "dtype", "kv_cache_memory_bytes")_QUANTIZATIONS = (None, "w4a16")-_DTYPES = {None: None, "bfloat16": torch.bfloat16}+_DTYPES = {None: None, "bfloat16": torch.bfloat16, "float16": torch.float16}# W4A16 runs on vLLM's Marlin kernel, the one whose exact output was verified.# vLLM tries Machete first on Hopper (SM90) GPUs, so it is disabled for the# engine; the loader then checks that every quantized layer uses Marlin._W4A16_KERNEL = "MarlinLinearKernel"_W4A16_DISABLED_KERNELS = ("MacheteLinearKernel",)-# Measured with the current checkpoint, with and without W4A16: in float16 the-# language model's residual stream passes float16's maximum (65,504) at layer 10-# and overflows, after which it emits no audio tokens.-_NO_FLOAT16 = ("this model's residual stream exceeds the float16 range (measured overflow "- "from layer 10); GPUs without bfloat16 (compute capability < 8.0) are not "- "supported by the vLLM backend")+# GPUs without bfloat16 (compute capability 7.5) can opt in to float16, a+# rescaled fallback with a precision change (fp16.py).+_FLOAT16_OPT_IN = ("To run on it anyway, opt in to the float16 fallback with "+ "vllm_options={'dtype': 'float16'}: it is supported but not the optimal "+ "path (a precision change from bfloat16; see the vLLM backend docs)")def resolve_options(options):@@ -83,10 +86,8 @@ def resolve_options(options):if kitten["quantization"] not in _QUANTIZATIONS:raise ValueError(f"vllm_options['quantization'] must be None or 'w4a16', "f"not {kitten['quantization']!r}")- if kitten["dtype"] == "float16":- raise ValueError(f"vllm_options['dtype']='float16' is not supported: {_NO_FLOAT16}")if kitten["dtype"] not in _DTYPES:- raise ValueError(f"vllm_options['dtype'] must be None or 'bfloat16', "+ raise ValueError(f"vllm_options['dtype'] must be None, 'bfloat16' or 'float16', "f"not {kitten['dtype']!r}")memory = kitten["kv_cache_memory_bytes"]if memory is not None and (isinstance(memory, bool) or not isinstance(memory, int)@@ -104,6 +105,10 @@ def check_options(options, weights):defaults, kitten = resolve_options(options)if kitten["quantization"] == "w4a16":check_w4a16_environment()+ if weights == "emb4" and kitten["dtype"] == "float16":+ # The float16 rescale was measured on the exact embedding (fp16.py).+ raise ValueError("vllm_options['dtype']='float16' cannot be combined with "+ "weights='emb4'; use weights='packed' or 'full'")return defaults, kitten@@ -145,9 +150,18 @@ def quantization_args(quantization):"hf_overrides": {"quantization_config": quantization_config()}}+def engine_args(quantization, float16):+ """quantization_args, plus the float16 rescale's model-config changes when asked."""+ args = quantization_args(quantization)+ if float16:+ from .fp16 import hf_overrides+ args["hf_overrides"] = {**args.get("hf_overrides", {}), **hf_overrides()}+ return args++class VLLMBackend:def __init__(self, owner, options=None):- validate_environment(owner.device)+ validate_environment(owner.device, dict(options or {}).get("dtype"))from transformers import AutoConfig, AutoTokenizerfrom ..speaker import load_speaker_head@@ -157,8 +171,10 @@ class VLLMBackend:default_dtype = (torch.bfloat16 if owner.config.get("dtype", "bf16") == "bf16"else torch.float32)dtype = _DTYPES[kitten["dtype"]] or default_dtype- if quantization == "w4a16" and dtype is not torch.bfloat16:- raise ValueError(f"W4A16 needs bfloat16 activations, not {dtype}")+ # Only an explicit opt-in runs the rescaled float16 model (fp16.py).+ float16 = dtype is torch.float16+ if quantization == "w4a16" and dtype not in (torch.bfloat16, torch.float16):+ raise ValueError(f"W4A16 needs bfloat16 or float16 activations, not {dtype}")self.max_model_len = defaults["max_model_len"]self.token_map = owner.token_maplm_dir = owner.repo_dir / owner.config.get("lm_dir", "lm")@@ -170,16 +186,28 @@ class VLLMBackend:# vLLM's own loader reads the full LM; this process only needs the# embedding and the speaker head, so do not expand the rest here.state = owner._load_lm_weights(lm_dir, include=_main_process_tensor)- self.embedding = state["model.embed_tokens.weight"].to(owner.dtype)- self.spk_proj = load_speaker_head(config.hidden_size, str(lm_dir),- owner.device, owner.dtype,- spk_dim=owner.config.get("speaker_embedding_dim", 512),- state=state)+ if float16:+ # Prompt embeddings at the engine's residual scale, built exactly as+ # the engine's own embedding table; the speaker head runs in float32.+ from .fp16 import ScaledSpeakerHead, scale_embedding+ self.embedding = scale_embedding(state["model.embed_tokens.weight"])+ self.spk_proj = ScaledSpeakerHead(load_speaker_head(+ config.hidden_size, str(lm_dir), owner.device, torch.float32,+ spk_dim=owner.config.get("speaker_embedding_dim", 512), state=state))+ else:+ self.embedding = state["model.embed_tokens.weight"].to(owner.dtype)+ self.spk_proj = load_speaker_head(config.hidden_size, str(lm_dir),+ owner.device, owner.dtype,+ spk_dim=owner.config.get("speaker_embedding_dim",+ 512),+ state=state)del stateloader_config = {"repo_dir": str(owner.repo_dir), "config": owner.config,"weights": owner.weights}if quantization is not None:loader_config["quantization"] = quantization+ if float16:+ loader_config["float16_rescale"] = Truewith _engine_environment(forced_environment):import vllm.envsfrom vllm import LLM@@ -196,7 +224,7 @@ class VLLMBackend:enable_prompt_embeds=True, enable_prefix_caching=False,logits_processors=["kittenml.kittentts2.vllm.logits:KittenLogitsProcessor"],logging_config=LoggingConfig(), disable_log_stats=True,- **quantization_args(quantization), **defaults)+ **engine_args(quantization, float16), **defaults)def prepare_request(self, prompt, speaker_embedding, settings, max_new_tokens, advanced):from vllm import SamplingParamsdiff --git a/kittenml/kittentts2/vllm/fp16.py b/kittenml/kittentts2/vllm/fp16.pynew file mode 100644index 0000000..95afb87--- /dev/null+++ b/kittenml/kittentts2/vllm/fp16.py@@ -0,0 +1,193 @@+"""Opt-in float16 mode for GPUs without bfloat16 (compute capability 7.5).++``vllm_options={"dtype": "float16"}`` is a fallback with a precision change:+the language model was trained and is published in bfloat16, whose range+float16 (largest value 65,504) does not have. Measured in bfloat16 over a+multilingual, long-form and cloning corpus, the residual stream (the running+sum every layer adds to) reaches 69,376 after layer 9 and 3,957,760 at the+final norm, and the MLP's inner activation (the ``down_proj`` input) reaches+154,624. Run as is in float16 these overflow and the model emits no speech.++The fix rescales the model exactly instead of changing what it computes:++- Every read of the residual stream goes through an RMSNorm, and+ ``RMSNorm(c * x)`` with epsilon ``c**2 * eps`` equals ``RMSNorm(x)`` with+ ``eps``. So the whole stream can be carried at ``2**-9`` of its value: the+ token embeddings and the speaker projection (the only inputs to the stream)+ are multiplied by ``2**-9``, so are the outputs of every attention and MLP+ branch, and the epsilon of the 57 norms that read the stream is multiplied+ by ``2**-18``. The attention and MLP inputs are normed, so they see the+ same values as in bfloat16.+- Each branch's ``2**-9`` is split over two weights, so that the activation+ inside the branch is scaled too: ``v_proj`` x ``2**-3`` and ``o_proj`` x+ ``2**-6`` (attention is linear in V), ``up_proj`` x ``2**-5`` and+ ``down_proj`` x ``2**-4`` (silu(gate) * up is linear in up). Multiplying by+ a power of two only changes the exponent, and every scaled weight of the+ published checkpoint is still exactly representable in float16 (checked at+ load time; the loader refuses otherwise).+- The output head shares the embedding matrix in the checkpoint. It is+ untied here and keeps the unscaled values, so the logits are not scaled+ either (the final norm already removes the stream's scale).++What does change is float16 rounding: activations and weights are stored+with float16's precision instead of bfloat16's, and the scaled embedding+(``2**-9`` of values around 0.03) partly falls into float16's subnormal+range, where 1.35% of its elements are rounded (worst error 4.6e-4 of a+token's RMS, a quarter of bfloat16's own rounding step). The untied head+costs one extra copy of the embedding (158,241 x 2,048, 618 MiB).++The scales were chosen from those measurements: ``2**-9`` leaves 8.5x+headroom below 65,504 at the largest value seen, and the MLP's inner+activation peaks at 4,832 (13.6x). They are specific to the published+checkpoint; another checkpoint needs new measurements.++This module does not import vLLM, so it can be tested on any machine.+"""++import re++import torch++# The residual stream is carried at 2**-RESIDUAL_SHIFT of its bfloat16 value.+RESIDUAL_SHIFT = 9+# Weight -> power-of-two shift (multiply by 2**-shift). Each branch's two+# shifts add up to RESIDUAL_SHIFT, so its output joins the stream at the+# stream's scale.+PROJECTION_SHIFTS = {"self_attn.v_proj": 3, "self_attn.o_proj": 6,+ "mlp.up_proj": 5, "mlp.down_proj": 4}+# RMSNorm(c x; c^2 eps) == RMSNorm(x; eps).+EPS_SHIFT = 2 * RESIDUAL_SHIFT++EMBEDDING = "model.embed_tokens.weight"+LM_HEAD = "lm_head.weight"++# The norms that read the residual stream. q_norm/k_norm read projections of+# an already normed input and keep their epsilon.+_RESIDUAL_NORM = re.compile(+ r"^model\.(?:layers\.\d+\.(?:input_layernorm|post_attention_layernorm)|norm)$")+# A scaled projection, dense (".weight") or as W4A16 group scales (".weight_scale").+_SCALED = re.compile(r"^model\.layers\.\d+\.(self_attn\.(?:v|o)_proj|mlp\.(?:up|down)_proj)"+ r"\.(weight|weight_scale)$")+# Every projection weight (dense) or W4A16 group scale: converted to float16+# here, so a lossy conversion is refused rather than left to vLLM's copy.+_PROJECTION = re.compile(r"^model\.layers\.\d+\.(?:self_attn\.(?:q|k|v|o)_proj|"+ r"mlp\.(?:gate|up|down)_proj)\.(?:weight|weight_scale)$")++# Written into the engine's model config (hf_overrides). vLLM keys its+# torch.compile cache on that config, so a float16 engine never reuses graphs+# compiled with another epsilon, and a change to this plan gets new ones.+PLAN = (f"residual 2^-{RESIDUAL_SHIFT}; "+ + ", ".join(f"{name.split('.')[-1]} 2^-{shift}"+ for name, shift in PROJECTION_SHIFTS.items())+ + f"; residual-norm eps x 2^-{EPS_SHIFT}; untied unscaled lm_head")+PLAN_KEY = "kittenml_float16_rescale"++if any(shift != RESIDUAL_SHIFT for shift in (+ PROJECTION_SHIFTS["self_attn.v_proj"] + PROJECTION_SHIFTS["self_attn.o_proj"],+ PROJECTION_SHIFTS["mlp.up_proj"] + PROJECTION_SHIFTS["mlp.down_proj"])):+ raise AssertionError("each branch must be scaled by exactly 2**-RESIDUAL_SHIFT")+++def hf_overrides():+ """Model-config changes for the engine: an untied head, and the plan."""+ return {"tie_word_embeddings": False, PLAN_KEY: PLAN}+++def shift_of(name):+ """The power-of-two shift applied to tensor `name` (0 when it is not scaled)."""+ if name == EMBEDDING:+ return RESIDUAL_SHIFT+ match = _SCALED.match(name)+ return PROJECTION_SHIFTS[match.group(1)] if match else 0+++def scaled(tensor, shift):+ """``tensor * 2**-shift`` in float16, rounded once (the scaling itself is exact)."""+ return (tensor.to(torch.float32) * 2.0 ** -shift).to(torch.float16)+++def _scaled_and_exact(tensor, shift):+ """(``scaled(tensor, shift)``, whether that float16 value is exact)."""+ target = tensor.to(torch.float32) * 2.0 ** -shift+ result = target.to(torch.float16)+ # Compare bits so a -0.0 that became +0.0 (or the reverse) is not exact.+ exact = torch.equal(result.to(torch.float32).view(torch.int32), target.view(torch.int32))+ return result, exact+++def is_exact(tensor, shift):+ """True when ``tensor * 2**-shift`` is exactly representable in float16."""+ return _scaled_and_exact(tensor, shift)[1]+++def exact(name, tensor, shift):+ """`scaled`, refusing a weight float16 cannot hold exactly at this scale."""+ result, ok = _scaled_and_exact(tensor, shift)+ if not ok:+ raise ValueError(f"{name} x 2^-{shift} is not exact in float16; the float16 rescale "+ "was measured for the published checkpoint and does not fit this one")+ return result+++def scale_embedding(weight):+ """Token embeddings as the float16 engine stores them (also used for prompts)."""+ return scaled(weight, RESIDUAL_SHIFT)+++def rescale_weights(weights):+ """Rescale (name, tensor) pairs, dense or W4A16, for the float16 engine.++ - ``model.embed_tokens.weight`` becomes the scaled embedding, and an+ unscaled float16 copy is added as ``lm_head.weight`` (the untied head);+ - every projection weight (dense) or W4A16 group scale becomes float16,+ with its shift for v/o/up/down, and must be exact;+ - everything else (norm weights, W4A16 codes and shapes) passes through+ unchanged, and vLLM converts floating-point tensors to float16 itself.+ """+ for name, tensor in weights:+ if name == LM_HEAD:+ raise ValueError("the float16 rescale expects a checkpoint whose output head is "+ "tied to the embedding, not a separate lm_head.weight")+ if name == EMBEDDING:+ yield name, scale_embedding(tensor)+ yield LM_HEAD, tensor.to(torch.float16)+ elif _PROJECTION.match(name):+ yield name, exact(name, tensor, shift_of(name))+ else:+ yield name, tensor+++class ScaledSpeakerHead(torch.nn.Module):+ """The speaker projection, run in float32, its output scaled and rounded to float16."""++ def __init__(self, head):+ super().__init__()+ self.head = head.float()++ def forward(self, speaker_embedding):+ return scaled(self.head(speaker_embedding.float()), RESIDUAL_SHIFT)+++def residual_norms(model):+ """(name, module) for every RMSNorm that reads the residual stream."""+ return [(name, module) for name, module in model.named_modules()+ if _RESIDUAL_NORM.match(name)]+++def scale_norm_epsilons(model, layers, eps):+ """Set the residual norms' epsilon to ``eps * 2**-EPS_SHIFT``; returns how many.++ Runs in the engine after the weights are loaded and before the model is+ compiled or captured. Setting (not multiplying) keeps a second call harmless.+ """+ norms = residual_norms(model)+ if len(norms) != 2 * layers + 1:+ raise RuntimeError(f"float16 rescale: expected {2 * layers + 1} residual-stream norms, "+ f"found {len(norms)}")+ target = eps * 2.0 ** -EPS_SHIFT+ for name, module in norms:+ if module.variance_epsilon not in (eps, target):+ raise RuntimeError(f"float16 rescale: {name} has epsilon {module.variance_epsilon}, "+ f"not the model's rms_norm_eps {eps}")+ module.variance_epsilon = target+ return len(norms)diff --git a/kittenml/kittentts2/vllm/loader.py b/kittenml/kittentts2/vllm/loader.pyindex 3d0fd06..3e8ecba 100644--- a/kittenml/kittentts2/vllm/loader.py+++ b/kittenml/kittentts2/vllm/loader.py@@ -13,9 +13,13 @@ class KittenModelLoader(BaseModelLoader):def load_weights(self, model, model_config):options = self.load_config.model_loader_extra_configquantization = options.get("quantization")+ # The opt-in float16 fallback for GPUs without bfloat16 (fp16.py).+ float16 = options.get("float16_rescale", False)+ if float16:+ _check_float16_model(model, model_config)if quantization == "w4a16":# vLLM chose each layer's kernel when it built the model; check before loading.- _check_w4a16_kernels(model, model_config)+ _check_w4a16_kernels(model, model_config, float16)# Exact 4-bit weights straight from the trits, one tensor at a time.from .w4a16 import iter_w4a16weights = iter_w4a16(options["repo_dir"], options["config"], options["weights"])@@ -25,10 +29,32 @@ class KittenModelLoader(BaseModelLoader):if not name.startswith("spk_proj."))else:raise ValueError(f"unknown KittenTTS quantization {quantization!r}")+ if float16:+ from .fp16 import rescale_weights+ weights = rescale_weights(weights)loaded = model.load_weights(weights)missing = set(dict(model.named_parameters())) - loaded - _kv_cache_scale_names(model)if missing:raise ValueError(f"weights missing from the checkpoint: {sorted(missing)[:5]}")+ if float16:+ from .fp16 import scale_norm_epsilons+ config = model_config.hf_config+ scale_norm_epsilons(model, config.num_hidden_layers, config.rms_norm_eps)+++def _check_float16_model(model, model_config):+ """The float16 rescale needs a float16 engine with its own, unscaled output head."""+ import torch++ refuse = "float16 rescale: refusing to load: "+ if model_config.dtype != torch.float16:+ raise RuntimeError(refuse + f"the engine dtype is {model_config.dtype}, not float16")+ head = getattr(getattr(model, "lm_head", None), "weight", None)+ embedding = getattr(getattr(getattr(model, "model", None), "embed_tokens", None),+ "weight", None)+ if getattr(model_config.hf_config, "tie_word_embeddings", True) or head is None \+ or head is embedding:+ raise RuntimeError(refuse + "the output head is tied to the scaled embedding")def _quantized_layers(model):@@ -39,7 +65,7 @@ def _quantized_layers(model):yield name, module, kernel-def _check_w4a16_kernels(model, model_config):+def _check_w4a16_kernels(model, model_config, float16=False):"""Every quantized linear layer must run exact W4A16 on Marlin.vLLM picks the mixed-precision kernel per GPU (Machete on Hopper, Marlin@@ -53,8 +79,11 @@ def _check_w4a16_kernels(model, model_config):from vllm.model_executor.kernels.linear import MarlinLinearKernelrefuse = "W4A16: refusing to load: "- if model_config.dtype != torch.bfloat16:- raise RuntimeError(refuse + f"the engine dtype is {model_config.dtype}, not bfloat16")+ # bfloat16, or float16 in the opt-in float16 mode (its group scales are rescaled).+ expected = torch.float16 if float16 else torch.bfloat16+ if model_config.dtype != expected:+ raise RuntimeError(refuse + f"the engine dtype is {model_config.dtype}, "+ f"not {str(expected).removeprefix('torch.')}")chosen, activations = {}, {}for name, _, kernel in _quantized_layers(model):chosen.setdefault(type(kernel).__name__, []).append(name)diff --git a/kittenml/kittentts2/vllm/w4a16.py b/kittenml/kittentts2/vllm/w4a16.pyindex 117e9d9..fe81ad2 100644--- a/kittenml/kittentts2/vllm/w4a16.py+++ b/kittenml/kittentts2/vllm/w4a16.py@@ -16,6 +16,10 @@ and the KV cache. ``spk_proj`` runs outside vLLM and is dropped here.The lossy ``emb4`` packing is refused: W4A16 is offered as an exactrepresentation of the published weights and is never combined with it.+In the opt-in float16 mode (fp16.py) the same codes are used; the group scales+of v/o/up/down are multiplied by that mode's powers of two and stored in+float16, which must be exact.+vLLM fuses q/k/v into ``qkv_proj`` and gate/up into ``gate_up_proj``(``Qwen3ForCausalLM.packed_modules_mapping``), so the 196 source projections(7 per layer) become 4 quantized modules per layer: 112 for the 28-layer model.diff --git a/tests/test_vllm.py b/tests/test_vllm.pyindex f0744e2..f116f6e 100644--- a/tests/test_vllm.py+++ b/tests/test_vllm.py@@ -306,15 +306,15 @@ _PROJECTIONS = {"self_attn.q_proj": (256, 256), "self_attn.k_proj": (128, 256),"mlp.down_proj": (256, 384)}-def _ternary(rows, cols, generator):+def _ternary(rows, cols, generator, low=1e-4):signs = torch.randint(-1, 2, (rows, cols), generator=generator)- scale = (torch.rand(rows, cols // 128, generator=generator) * 0.5 + 1e-4).to(torch.bfloat16)+ scale = (torch.rand(rows, cols // 128, generator=generator) * 0.5 + low).to(torch.bfloat16)scale[0, 0] = 0 # an all-zero group has a zero scalesigns[0, :128] = 0return signs.to(torch.bfloat16) * scale.repeat_interleave(128, dim=1)-def _w4a16_checkpoint(tmp_path, layers=2, perturb=None):+def _w4a16_checkpoint(tmp_path, layers=2, perturb=None, low=1e-4):"""A small Qwen3-shaped LM written as both full bf16 and packed ternary."""generator = torch.Generator().manual_seed(0)state = {"model.embed_tokens.weight": torch.randn(32, 256, generator=generator).to(torch.bfloat16),@@ -328,7 +328,7 @@ def _w4a16_checkpoint(tmp_path, layers=2, perturb=None):state[prefix + "input_layernorm.weight"] = torch.ones(256, dtype=torch.bfloat16)state[prefix + "self_attn.q_norm.weight"] = torch.ones(128, dtype=torch.bfloat16)for name, shape in _PROJECTIONS.items():- state[prefix + name + ".weight"] = _ternary(*shape, generator)+ state[prefix + name + ".weight"] = _ternary(*shape, generator, low)if perturb is not None:weight = state[perturb]weight[1, 5] = weight[1, :128].abs().max() / 2 # no longer ternary in that group@@ -460,6 +460,8 @@ def test_w4a16_accepts_emb4_and_packs_its_ternary_body_exactly(tmp_path):assert torch.equal(out[key], value)check_options({"quantization": "w4a16"}, "emb4")+ with pytest.raises(ValueError, match="float16"):+ check_options({"dtype": "float16"}, "emb4")def test_w4a16_targets_112_fused_modules_for_196_projections():@@ -510,31 +512,39 @@ def test_vllm_options_validate_quantization_dtype_and_cache_size():resolve_options(bad)-def test_float16_is_refused_with_the_measured_reason():- # float16 was measured to overflow at layer 10 with the current checkpoint- # (with and without W4A16) and then produce no audio, so it is refused.+def test_float16_is_an_explicit_option():+ # float16 is the opt-in fallback for GPUs without bfloat16 (fp16.py); it is+ # never chosen unless asked for.from kittenml.kittentts2.vllm.backend import resolve_optionsfor options in ({"dtype": "float16"}, {"dtype": "float16", "quantization": "w4a16"}):- with pytest.raises(ValueError, match=r"'float16' is not supported: this model's residual "- r"stream exceeds the float16 range \(measured overflow "- r"from layer 10\).*compute capability < 8.0"):- resolve_options(options)+ engine, kitten = resolve_options(options)+ assert kitten["dtype"] == "float16" and "dtype" not in engine+ assert resolve_options(None)[1]["dtype"] is None-def test_dtype_follows_hardware_and_gpus_without_bfloat16_are_refused():+def test_dtype_follows_hardware_and_gpus_without_bfloat16_need_the_float16_opt_in():from kittenml.kittentts2.vllm.backend import validate_environmentwith mock.patch("kittenml.kittentts2.vllm.backend.importlib.util.find_spec", return_value=object()), \mock.patch("torch.cuda.is_available", return_value=True):for capability in ((8, 0), (8, 6), (12, 0)):with mock.patch("torch.cuda.get_device_capability", return_value=capability):- validate_environment()- for capability, name in (((7, 5), "SM75"), ((7, 0), "SM70")):+ for dtype in (None, "bfloat16", "float16"):+ validate_environment(dtype=dtype)+ with mock.patch("torch.cuda.get_device_capability", return_value=(7, 5)):+ for dtype in (None, "bfloat16"):+ with pytest.raises(RuntimeError, match=r"SM75\) has no bfloat16.*opt in to the "+ r"float16 fallback with vllm_options="+ r"\{'dtype': 'float16'\}.*not the optimal"):+ validate_environment(dtype=dtype)+ validate_environment(dtype="float16")+ for capability, name in (((7, 0), "SM70"), ((6, 1), "SM61")):with mock.patch("torch.cuda.get_device_capability", return_value=capability):- with pytest.raises(RuntimeError, match=name + ".*no bfloat16.*float16 range"- ".*layer 10.*< 8.0"):- validate_environment()+ for dtype in (None, "float16"):+ with pytest.raises(RuntimeError, match=name + r"\) is not supported by the "+ r"vLLM backend.*7.5 or newer"):+ validate_environment(dtype=dtype)def test_vllm_options_are_checked_before_model_download(monkeypatch):@@ -543,7 +553,8 @@ def test_vllm_options_are_checked_before_model_download(monkeypatch):mock.patch("kittenml.kittentts2.vllm.backend.validate_environment"), \mock.patch("kittenml.get_model.snapshot_download") as download:for options, weights, match in [({"quantisation": "w4a16"}, None, "unknown vllm_options"),- ({"dtype": "float16"}, None, "float16 range")]:+ ({"dtype": "float16"}, "emb4", "float16.*emb4"),+ ({"dtype": "fp16"}, None, "'float16'")]:with pytest.raises(ValueError, match=match):KittenTTS("example/repo", backend="vllm", weights=weights, vllm_options=options)monkeypatch.setenv("VLLM_MARLIN_INPUT_DTYPE", "int8")@@ -559,9 +570,13 @@ def test_gpus_without_bfloat16_are_refused_before_model_download():mock.patch("torch.cuda.is_available", return_value=True), \mock.patch("torch.cuda.get_device_capability", return_value=(7, 5)), \mock.patch("kittenml.get_model.snapshot_download") as download:- with pytest.raises(RuntimeError, match="SM75.*no bfloat16"):+ with pytest.raises(RuntimeError, match="SM75.*no bfloat16.*'dtype': 'float16'"):KittenTTS("example/repo", backend="vllm")download.assert_not_called()+ # With the opt-in, the checks pass and the download starts.+ download.side_effect = RuntimeError("download started")+ with pytest.raises(RuntimeError, match="download started"):+ KittenTTS("example/repo", backend="vllm", vllm_options={"dtype": "float16"})def test_w4a16_refuses_marlin_environment_overrides(monkeypatch, tmp_path):@@ -653,7 +668,7 @@ def test_engine_receives_w4a16_arguments_and_main_process_loads_only_conditioninos.environ.pop("VLLM_DISABLED_KERNELS", None)owner, kwargs = run(None)assert {"quantization", "hf_overrides", "kv_cache_memory_bytes"}.isdisjoint(kwargs)- assert "quantization" not in kwargs["model_loader_extra_config"]+ assert set(kwargs["model_loader_extra_config"]) == {"repo_dir", "config", "weights"}assert kwargs["dtype"] is torch.bfloat16 and owner.dtype is torch.bfloat16assert kwargs["disabled_kernels"] is Noneowner, kwargs = run({"quantization": "w4a16"})@@ -663,8 +678,11 @@ def test_engine_receives_w4a16_arguments_and_main_process_loads_only_conditioninowner, kwargs = run({"quantization": "w4a16", "kv_cache_memory_bytes": 1 << 30,"dtype": "bfloat16"})assert kwargs["quantization"] == "compressed-tensors"+ # bfloat16 W4A16: only the quantization config, the head stays tied.+ assert set(kwargs["hf_overrides"]) == {"quantization_config"}assert kwargs["hf_overrides"]["quantization_config"]["ignore"] == ["lm_head"]assert kwargs["model_loader_extra_config"]["quantization"] == "w4a16"+ assert "float16_rescale" not in kwargs["model_loader_extra_config"]assert kwargs["kv_cache_memory_bytes"] == 1 << 30assert kwargs["dtype"] is torch.bfloat16 and owner.dtype is torch.bfloat16@@ -857,3 +875,321 @@ def test_w4a16_exactness_checks_compare_bits_not_values(tmp_path):save_file({"model.layers.0.self_attn.o_proj.weight": weight},tmp_path / "lm" / "model.safetensors")assert len(list(iter_w4a16(tmp_path, {}, "full"))) == 3+++# ββ Opt-in float16 (vllm_options={"dtype": "float16"}), fp16.py βββββββββββββββββ++def _qwen3(layers=2, tied=True):+ from transformers import Qwen3Config, Qwen3ForCausalLM++ config = Qwen3Config(vocab_size=32, hidden_size=256, intermediate_size=384,+ num_hidden_layers=layers, num_attention_heads=2, num_key_value_heads=1,+ head_dim=128, rms_norm_eps=1e-6, tie_word_embeddings=tied,+ max_position_embeddings=64)+ config._attn_implementation = "eager"+ return Qwen3ForCausalLM(config).double().eval()+++def _realistic_state(model, generator):+ """Random weights at the published model's scales: small embeddings (mean(x^2)+ around 1e-3, so an unscaled epsilon would matter) and a residual stream that grows."""+ state = {}+ for name, tensor in model.state_dict().items():+ if name == "lm_head.weight":+ continue+ if name.endswith("norm.weight"):+ value = torch.rand(tensor.shape, generator=generator, dtype=torch.float64) + 0.5+ elif name == "model.embed_tokens.weight":+ value = torch.randn(tensor.shape, generator=generator, dtype=torch.float64) * 0.033+ else:+ value = torch.randn(tensor.shape, generator=generator, dtype=torch.float64) * 0.5+ state[name] = value+ return state+++def test_float16_rescale_is_the_same_model_in_float64():+ # The rescale is exact: with the stream's inputs at 2^-9, every branch at 2^-9 and+ # the residual norms' epsilon at 2^-18, the logits equal the original model's.+ # Checked on transformers' Qwen3 in float64, with the plan's own functions.+ from kittenml.kittentts2.vllm.fp16 import (EMBEDDING, LM_HEAD, RESIDUAL_SHIFT,+ residual_norms, scale_norm_epsilons, shift_of)++ generator = torch.Generator().manual_seed(3)+ reference = _qwen3(tied=True)+ state = _realistic_state(reference, generator)+ reference.load_state_dict(state, strict=False)+ reference.tie_weights()+ rescaled = _qwen3(tied=False)+ rescaled.load_state_dict({**{name: tensor * 2.0 ** -shift_of(name)+ for name, tensor in state.items()},+ LM_HEAD: state[EMBEDDING]}, strict=True)+ assert rescaled.lm_head.weight.data_ptr() != rescaled.model.embed_tokens.weight.data_ptr()+ assert scale_norm_epsilons(rescaled, layers=2, eps=1e-6) == 5+ assert [name for name, _ in residual_norms(rescaled)] == [+ "model.layers.0.input_layernorm", "model.layers.0.post_attention_layernorm",+ "model.layers.1.input_layernorm", "model.layers.1.post_attention_layernorm",+ "model.norm"]+ assert rescaled.model.layers[0].self_attn.q_norm.variance_epsilon == 1e-6++ ids = torch.tensor([[0, 5, 9, 2, 31, 7]])+ embeds = state[EMBEDDING][ids].clone()+ embeds[0, 0] = torch.randn(256, generator=generator, dtype=torch.float64) # spk_proj slot+ with torch.no_grad():+ expected = reference(inputs_embeds=embeds).logits+ actual = rescaled(inputs_embeds=embeds * 2.0 ** -RESIDUAL_SHIFT).logits+ # transformers' RMSNorm computes in float32, so agreement is to float32 rounding.+ torch.testing.assert_close(actual, expected, rtol=1e-5, atol=1e-5)+ # Negative control: without the epsilon scale the first norm (embeddings at+ # 2^-9, mean(x^2) ~ 4e-9 << 1e-6) is visibly wrong.+ for _, module in residual_norms(rescaled):+ module.variance_epsilon = 1e-6+ wrong = rescaled(inputs_embeds=embeds * 2.0 ** -RESIDUAL_SHIFT).logits+ assert (wrong - expected).abs().max() > 1e-2+++def test_float16_plan_shifts_each_branch_by_the_residual_scale():+ from kittenml.kittentts2.vllm import fp16++ p = "model.layers.3."+ shifts = {name: fp16.shift_of(p + name + ".weight") for name in _PROJECTIONS}+ assert shifts == {"self_attn.q_proj": 0, "self_attn.k_proj": 0, "self_attn.v_proj": 3,+ "self_attn.o_proj": 6, "mlp.gate_proj": 0, "mlp.up_proj": 5,+ "mlp.down_proj": 4}+ assert fp16.shift_of(p + "mlp.down_proj.weight_scale") == 4+ assert fp16.shift_of(p + "mlp.down_proj.weight_packed") == 0+ assert fp16.shift_of("model.embed_tokens.weight") == fp16.RESIDUAL_SHIFT == 9+ assert fp16.shift_of(p + "input_layernorm.weight") == 0+ assert fp16.EPS_SHIFT == 18+ assert fp16.hf_overrides() == {"tie_word_embeddings": False,+ "kittenml_float16_rescale": fp16.PLAN}+ assert fp16.PLAN == ("residual 2^-9; v_proj 2^-3, o_proj 2^-6, up_proj 2^-5, down_proj 2^-4; "+ "residual-norm eps x 2^-18; untied unscaled lm_head")+++def test_float16_plan_reaches_the_config_vllm_hashes_for_its_compile_cache():+ # vLLM's ModelConfig.compute_hash ignores hf_overrides itself but hashes+ # hf_config.to_json_string() after the overrides are applied with setattr.+ from transformers import Qwen3Config+ from kittenml.kittentts2.vllm.fp16 import PLAN, hf_overrides++ config = Qwen3Config(tie_word_embeddings=True)+ before = config.to_json_string()+ for key, value in hf_overrides().items():+ setattr(config, key, value)+ after = config.to_json_string()+ assert after != before and PLAN in after and config.tie_word_embeddings is False++[email protected]("variant", ["packed", "full"])+def test_float16_rescale_dense_and_w4a16_give_the_same_exact_float16_weights(tmp_path, variant):+ from kittenml.kittentts2.vllm.fp16 import rescale_weights, scaled, shift_of+ from kittenml.kittentts2.vllm.w4a16 import bit_equal, is_projection, iter_w4a16++ # Group scales >= 2^-6, as in the published checkpoint: every shifted weight+ # stays a normal float16 number.+ state, config = _w4a16_checkpoint(tmp_path, low=2.0 ** -6)+ weights = load_lm_weights(tmp_path, config, variant)+ dense = dict(rescale_weights((n, t) for n, t in weights.items()+ if not n.startswith("spk_proj.")))+ original = dict(iter_w4a16(tmp_path, config, variant))+ quantized = dict(rescale_weights(original.items()))+ unpack = _reference_unpack()+ embedding = state["model.embed_tokens.weight"]+ for out in (dense, quantized):+ assert bit_equal(out["model.embed_tokens.weight"], scaled(embedding, 9))+ assert bit_equal(out["lm_head.weight"], embedding.to(torch.float16)) # untied, unscaled+ assert bit_equal(out["model.norm.weight"], state["model.norm.weight"]) # unchanged+ for name in (n for n in state if is_projection(n)):+ exact = state[name].to(torch.float32) * 2.0 ** -shift_of(name)+ assert dense[name].dtype == torch.float16+ assert torch.equal(dense[name].to(torch.float32), exact), name+ base = name.removesuffix(".weight")+ scale = quantized[base + ".weight_scale"]+ assert scale.dtype == torch.float16+ assert quantized[base + ".weight_packed"] is original[base + ".weight_packed"]+ assert torch.equal(scale.to(torch.float32), original[base + ".weight_scale"].float()+ * 2.0 ** -shift_of(name))+ signs = unpack(quantized[base + ".weight_packed"], state[name].shape)+ rebuilt = signs.to(torch.float16) * scale.repeat_interleave(128, dim=1)+ assert bit_equal(rebuilt, dense[name]), name+++def test_float16_rescale_refuses_weights_it_cannot_hold_exactly():+ from kittenml.kittentts2.vllm.fp16 import is_exact, rescale_weights++ weight = torch.full((2, 128), 0.25, dtype=torch.bfloat16)+ weight[1, 7] = 2.0 ** -20 # x 2^-6 = 2^-26: below float16's smallest subnormal+ pairs = [("model.layers.0.self_attn.o_proj.weight", weight)]+ with pytest.raises(ValueError, match=r"o_proj.weight x 2\^-6 is not exact in float16"):+ dict(rescale_weights(pairs))+ # The same value is exact without a shift (q_proj) ... and -0.0 must stay -0.0.+ assert len(dict(rescale_weights([("model.layers.0.self_attn.q_proj.weight", weight)]))) == 1+ zero = torch.tensor([-0.0], dtype=torch.bfloat16)+ assert is_exact(zero, 6) and is_exact(torch.tensor([65504.0]), 0)+ assert not is_exact(torch.tensor([65536.0]), 0) # overflow is not exact+ with pytest.raises(ValueError, match="tied"):+ dict(rescale_weights([("lm_head.weight", weight)]))+++def test_float16_prompt_embeddings_match_the_engine_scale():+ pytest.importorskip("vllm")+ from kittenml.kittentts2.vllm.backend import VLLMBackend+ from kittenml.kittentts2.vllm.fp16 import ScaledSpeakerHead, scale_embedding, scaled++ generator = torch.Generator().manual_seed(5)+ table = (torch.randn(16, 4, generator=generator) * 0.03).to(torch.bfloat16)+ head = torch.nn.Linear(4, 4).to(torch.bfloat16)+ backend = object.__new__(VLLMBackend)+ backend.max_model_len = 32+ backend.token_map = TokenMap(5, 10, 1, 2, 3, 4, 0, 15)+ backend.embedding = scale_embedding(table)+ backend.spk_proj = ScaledSpeakerHead(head)+ speaker = torch.randn(1, 4, generator=generator).to(torch.float16)+ settings = dict(temperature=0.7, top_k=5, top_p=0.8, min_p=0.02)+ advanced = dict(seed=1, repetition_penalty=1.2, repetition_window=3,+ token_run_penalty=1.5, token_run_grace=4)+ request, _ = backend.prepare_request([0, 3, 1], speaker, settings, 8, advanced)+ embeds = request["prompt_embeds"]+ assert embeds.dtype == torch.float16+ assert torch.equal(embeds[1:], scaled(table[[3, 1]], 9))+ expected = scaled(torch.nn.functional.linear(speaker.float(), head.weight.float(),+ head.bias.float()), 9).squeeze(0)+ assert torch.equal(embeds[0], expected)+++def _norm(eps=1e-6):+ module = torch.nn.Module()+ module.variance_epsilon = eps+ return module+++def _float16_engine_model(layers=2, tied=False):+ """Just enough of vLLM's Qwen3ForCausalLM for the loader's float16 checks."""+ model = torch.nn.Module()+ model.model = torch.nn.Module()+ model.model.embed_tokens = torch.nn.Embedding(4, 2)+ model.model.layers = torch.nn.ModuleList()+ for _ in range(layers):+ layer = torch.nn.Module()+ layer.input_layernorm, layer.post_attention_layernorm = _norm(), _norm()+ layer.self_attn = torch.nn.Module()+ layer.self_attn.q_norm, layer.self_attn.k_norm = _norm(), _norm()+ model.model.layers.append(layer)+ model.model.norm = _norm()+ model.lm_head = model.model.embed_tokens if tied else torch.nn.Embedding(4, 2)+ model.loaded = {}++ def load_weights(weights):+ model.loaded.update(weights)+ return set(dict(model.named_parameters()))+ model.load_weights = load_weights+ return model+++def test_vllm_loader_float16_rescales_weights_and_residual_norm_epsilon_only():+ pytest.importorskip("vllm")+ from kittenml.kittentts2.vllm.loader import KittenModelLoader++ loader = object.__new__(KittenModelLoader)+ loader.load_config = SimpleNamespace(model_loader_extra_config={+ "repo_dir": "unused", "config": {}, "weights": "packed", "float16_rescale": True})+ hf_config = SimpleNamespace(num_hidden_layers=2, rms_norm_eps=1e-6, tie_word_embeddings=False)+ fp16_config = SimpleNamespace(dtype=torch.float16, hf_config=hf_config)+ embedding = torch.full((4, 2), 0.03, dtype=torch.bfloat16)+ state = {"model.embed_tokens.weight": embedding, "spk_proj.0.bias": torch.zeros(2),+ "model.layers.0.mlp.up_proj.weight": torch.full((2, 2), 0.5, dtype=torch.bfloat16)}+ model = _float16_engine_model()+ with mock.patch("kittenml.kittentts2.vllm.loader.load_lm_weights", return_value=state):+ loader.load_weights(model, fp16_config)+ assert set(model.loaded) == {"model.embed_tokens.weight", "lm_head.weight",+ "model.layers.0.mlp.up_proj.weight"}+ assert torch.equal(model.loaded["lm_head.weight"], embedding.to(torch.float16))+ assert model.loaded["model.layers.0.mlp.up_proj.weight"].unique().tolist() == [2.0 ** -6]+ for layer in model.model.layers:+ assert layer.input_layernorm.variance_epsilon == 1e-6 * 2.0 ** -18+ assert layer.post_attention_layernorm.variance_epsilon == 1e-6 * 2.0 ** -18+ assert layer.self_attn.q_norm.variance_epsilon == 1e-6 # reads a normed input+ assert layer.self_attn.k_norm.variance_epsilon == 1e-6+ assert model.model.norm.variance_epsilon == 1e-6 * 2.0 ** -18+ loader.load_weights(model, fp16_config) # a second load does not scale twice+ assert model.model.norm.variance_epsilon == 1e-6 * 2.0 ** -18+ with pytest.raises(RuntimeError, match="engine dtype is torch.bfloat16, not float16"):+ loader.load_weights(_float16_engine_model(),+ SimpleNamespace(dtype=torch.bfloat16, hf_config=hf_config))+ with pytest.raises(RuntimeError, match="tied to the scaled embedding"):+ loader.load_weights(_float16_engine_model(tied=True), fp16_config)+ with pytest.raises(RuntimeError, match="expected 5 residual-stream norms, found 7"):+ loader.load_weights(_float16_engine_model(layers=3), fp16_config)+++def test_vllm_loader_bfloat16_paths_hand_over_the_checkpoint_tensors_untouched():+ # Without the float16 opt-in the loader passes the very same tensors and never+ # touches an epsilon, for the dense and the W4A16 path.+ pytest.importorskip("vllm")+ from vllm.model_executor.kernels.linear import MarlinLinearKernel+ from kittenml.kittentts2.vllm.loader import KittenModelLoader++ bf16 = SimpleNamespace(dtype=torch.bfloat16)+ for quantization in (None, "w4a16"):+ loader = object.__new__(KittenModelLoader)+ extra = {"repo_dir": "unused", "config": {}, "weights": "packed"}+ if quantization:+ extra["quantization"] = quantization+ loader.load_config = SimpleNamespace(model_loader_extra_config=extra)+ model = _float16_engine_model(tied=True)+ model.proj = _linear_with_kernel(MarlinLinearKernel)+ tensors = {"model.embed_tokens.weight": torch.full((4, 2), 0.03, dtype=torch.bfloat16),+ "model.layers.0.mlp.up_proj.weight": torch.ones(2, 2, dtype=torch.bfloat16)}+ with mock.patch("kittenml.kittentts2.vllm.loader.load_lm_weights", return_value=tensors), \+ mock.patch("kittenml.kittentts2.vllm.w4a16.iter_w4a16",+ return_value=iter(tensors.items())):+ loader.load_weights(model, bf16)+ assert set(model.loaded) == set(tensors)+ assert all(model.loaded[name] is tensors[name] for name in tensors)+ assert all(module.variance_epsilon == 1e-6 for name, module in model.named_modules()+ if hasattr(module, "variance_epsilon"))+++def test_engine_receives_the_float16_rescale_only_when_asked(tmp_path):+ pytest.importorskip("vllm")+ from kittenml.kittentts2.vllm.backend import VLLMBackend+ from kittenml.kittentts2.vllm.fp16 import PLAN, ScaledSpeakerHead, scale_embedding++ seen = []+ embedding = torch.full((2, 4), 0.03, dtype=torch.bfloat16)++ def run(options):+ owner = SimpleNamespace(device="cpu", repo_dir=tmp_path, config={}, weights="packed",+ token_map=None, _load_lm_weights=mock.Mock(return_value={+ "model.embed_tokens.weight": embedding}))+ with mock.patch("kittenml.kittentts2.vllm.backend.validate_environment") as validate, \+ mock.patch("transformers.AutoConfig.from_pretrained", return_value=SimpleNamespace(+ model_type="qwen3", hidden_size=4)), \+ mock.patch("transformers.AutoTokenizer.from_pretrained"), \+ mock.patch("kittenml.kittentts2.speaker.load_speaker_head",+ return_value=torch.nn.Linear(4, 4)) as head, \+ mock.patch("vllm.LLM", side_effect=lambda **kwargs: seen.append(kwargs)):+ backend = VLLMBackend(owner, options)+ validate.assert_called_once_with("cpu", (options or {}).get("dtype"))+ return owner, backend, head, seen[-1]++ for options in ({"dtype": "float16"}, {"dtype": "float16", "quantization": "w4a16"}):+ owner, backend, head, kwargs = run(options)+ assert kwargs["dtype"] is torch.float16 and owner.dtype is torch.float16+ assert kwargs["model_loader_extra_config"]["float16_rescale"] is True+ overrides = kwargs["hf_overrides"]+ assert overrides["tie_word_embeddings"] is False+ assert overrides["kittenml_float16_rescale"] == PLAN+ assert ("quantization_config" in overrides) == ("quantization" in options)+ assert torch.equal(backend.embedding, scale_embedding(embedding))+ assert isinstance(backend.spk_proj, ScaledSpeakerHead)+ assert head.call_args.args[3] is torch.float32 # the speaker head runs in float32+ for options in (None, {"dtype": "bfloat16"}, {"quantization": "w4a16"}):+ owner, backend, head, kwargs = run(options)+ assert kwargs["dtype"] is torch.bfloat16 and owner.dtype is torch.bfloat16+ assert "float16_rescale" not in kwargs["model_loader_extra_config"]+ assert "tie_word_embeddings" not in kwargs.get("hf_overrides", {})+ assert backend.embedding.dtype is torch.bfloat16 and torch.equal(backend.embedding,+ embedding)+ assert backend.spk_proj is head.return_value+ assert head.call_args.args[3] is torch.bfloat16