Open-source State-of-the-art TTS model which runs on a CPU π»
Python
Add exact 4-bit (W4A16) weights to the vLLM backend
haaziq-sl47e02b9a002c
diff --git a/README.md b/README.mdindex 0667e9a..0b9c897 100644--- a/README.md+++ b/README.md@@ -213,7 +213,8 @@ print(model.last_report) # Generation timings and token IDs.## Running with vLLM-For GPU inference with vLLM on Linux and an NVIDIA CUDA GPU, install the optional backend:+For GPU inference with vLLM on Linux and an NVIDIA CUDA GPU with compute capability+8.0 or newer (Ampere or later, for bfloat16), install the optional backend:```bashpip install "kittenml[vllm]"@@ -225,7 +226,8 @@ Select `backend="vllm"`:from kittenml import KittenTTSdef main():- m = KittenTTS("KittenML/kitten-tts-2", backend="vllm")+ m = KittenTTS("KittenML/kitten-tts-2", backend="vllm",+ vllm_options={"quantization": "w4a16"})m.generate_to_file("Hello there.", "output.wav", voice="Bruno")print("Saved output.wav")@@ -234,7 +236,10 @@ if __name__ == "__main__":```Keep the main guard when running a script: vLLM starts a worker process. The same-voices, decoders, weight choices and generation arguments apply. See the+voices, decoders, weight choices and generation arguments apply. The example+uses 4-bit language-model weights (`"quantization": "w4a16"`): the same weights, exactly, in about+2 GB less GPU memory, and faster for single requests. Leave out `vllm_options` to run in bf16; see+[4-bit weights](docs/vllm.md#4-bit-weights). See the[vLLM guide](docs/vllm.md) for memory settings and benchmarks.diff --git a/docs/vllm.md b/docs/vllm.mdindex b0d32c6..334f594 100644--- a/docs/vllm.md+++ b/docs/vllm.md@@ -1,72 +1,103 @@# Running with vLLM-The optional `vllm` backend runs KittenTTS 2's language model on an NVIDIA CUDA-GPU. Audio decoding, voice cloning and text preparation use the existing-KittenTTS implementations. The default backend remains PyTorch and already-supports CUDA.--Install the optional backend and select `backend="vllm"` as shown in the-[README](../README.md#running-with-vllm). This backend requires Linux and uses-vLLM 0.31.0; see its [GPU requirements](https://docs.vllm.ai/en/v0.31.0/getting_started/installation/gpu/)-for compatible NVIDIA drivers.-It loads the selected published weights directly; no manual export-or additional language-model checkpoint is required. Packed weights are expanded in-memory, so their smaller download size does not reduce GPU memory use.--The engine defaults to a 4,096-token context, up to eight active sequences, and-half the GPU memory, leaving space for audio decoding. In the README example,-pass `vllm_options` when constructing the model to change these settings:--```python-m = KittenTTS("KittenML/kitten-tts-2", backend="vllm",- vllm_options={"gpu_memory_utilization": 0.5,- "max_model_len": 4096})+The optional `vllm` backend runs KittenTTS 2's language model with vLLM 0.31.0 on an NVIDIA GPU;+audio decoding, voice cloning and text preparation stay the same. It needs Linux and a GPU with+compute capability 8.0 or newer (Ampere or later); older GPUs are refused before any download.++Install and select it as shown in the [README](../README.md#running-with-vllm). That install gets+CUDA 13.0 builds of vLLM and PyTorch, which need a driver for CUDA 13.0+([GPU requirements](https://docs.vllm.ai/en/v0.31.0/getting_started/installation/gpu/)). If+`nvidia-smi` reports a lower CUDA version, install the CUDA 12.9 builds with+[uv](https://docs.astral.sh/uv/) instead (checked with driver 570.195, which reports CUDA 12.8):++```bash+curl -LO https://github.com/vllm-project/vllm/releases/download/v0.31.0/vllm-0.31.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl+uv pip install --torch-backend cu129 "kittenml[vllm]" ./vllm-0.31.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl```-`gpu_memory_utilization` is a fraction of the GPU's total VRAM. It sets vLLM's-budget; the audio decoder and other GPU processes need additional memory.+## Options++| Option | Default | What it is for |+|---|---|---|+| `gpu_memory_utilization` | `0.5` | Share of the GPU's memory vLLM may use for the language model and its cache. Audio decoding needs memory outside this share. |+| `max_model_len` | `4096` | Longest request in tokens: the voice's reference prompt and the text, plus `max_new_tokens` (1000 by default). A request that does not fit raises an error. |+| `max_num_seqs` | `8` | Most requests generated at the same time. |+| `kv_cache_memory_bytes` | not set | A fixed cache size in bytes, used instead of sizing the cache from `gpu_memory_utilization`. |+| `enforce_eager` | `False` | Turns off CUDA graphs. Keep it off: with it on, 4-bit can be slower than the default bf16. |+| `quantization` | not set | `"w4a16"` for 4-bit weights; see [4-bit weights](#4-bit-weights). |+| `dtype` | `"bfloat16"` | Precision of the language model. Only `"bfloat16"` is accepted; `"float16"` is refused. |++Pass these as `vllm_options` when constructing the model, for example+`vllm_options={"max_model_len": 2048}`; unknown option names are refused before any download. Select+the GPU with `CUDA_VISIBLE_DEVICES`; set `VLLM_LOGGING_LEVEL=INFO` before starting Python to see+routine engine logs. Long text uses the default backend's sentence chunking, and `generate_stream`+yields each audio chunk once it is complete. The same seed can give different audio across backends+and with or without 4-bit weights. Start-up is slow (see [Benchmarks](#benchmarks)), so reuse the+loaded model.-If VRAM is limited, reduce `max_model_len` to lower cache requirements. If startup-reports no available memory for cache blocks, increase `gpu_memory_utilization`-to give vLLM a larger share of VRAM, leaving space for audio decoding and other-GPU processes.+## Low GPU memory-Routine engine logs are quiet by default; warnings and errors remain visible.-Set `VLLM_LOGGING_LEVEL=INFO` before starting Python to see startup diagnostics.+- Reduce `max_model_len`, keeping room for the prompt plus `max_new_tokens`: a shorter context+ needs a smaller cache.+- If startup reports no memory available for the cache, raise `gpu_memory_utilization`, leaving room+ for audio decoding, or set `kv_cache_memory_bytes`. The cache needs 112 KiB per token; a cache of+ exactly `max_model_len` tokens is rejected, so leave some margin.+- Use [4-bit weights](#4-bit-weights); the smaller `weights="packed"` download alone does not+ lower GPU memory.+- Cloning without `reference_text` also loads Whisper large-v3 on the GPU to transcribe the clip;+ pass `reference_text` to skip it.-The context must fit the reference prompt and the requested generation budget.-Long text still uses KittenTTS's sentence chunking. `generate_stream` yields-completed audio chunks, just like the default backend.+## 4-bit weights-`vllm_options` also accepts `max_num_seqs` and `enforce_eager`. Select a GPU with-`CUDA_VISIBLE_DEVICES` before starting Python. Sampling implementations differ-between the backends, so a matching seed does not guarantee identical audio.+Add `"quantization": "w4a16"` to `vllm_options` to use about 2 GB less GPU memory for the language+model's weights (1.29 instead of 3.24 GiB). The model's weights stay exactly the same, and measured+audio quality was the same as without it. With default settings vLLM still takes half the GPU and+gives the saved memory to its cache; to lower total use, also set `kv_cache_memory_bytes` or a lower+`gpu_memory_utilization`. It works with `weights="packed"` (the default) or `"full"`, and also+with `weights="emb4"`. That one is lossy: its embedding is stored in 4 bits (at export, perplexity+13.60 against 13.50), and it only makes the download smaller, not GPU memory, because the+embedding is expanded to bf16 on the GPU either way. KittenTTS checks that vLLM runs the 4-bit layers+with the kernel they were verified with, and refuses settings that would change it.++The biggest gain is with one request at a time: the language model ran about 1.4x faster on an RTX+PRO 6000 (96 GB) and 1.8x faster on an RTX A2000 (6 GB). With many concurrent requests the gain is+small (1.02x with 50 at once on the RTX PRO 6000). On the RTX A2000, bf16 does not start with+default `vllm_options` but 4-bit does; with `kv_cache_memory_bytes` 512 MiB and `max_num_seqs` 1 for+both, five sentences one at a time:++| RTX A2000 (6 GB) | bf16 (default) | 4-bit |+|---|---:|---:|+| Peak GPU memory, whole card | 6,116 MiB | 4,096 MiB |+| Median time per sentence | 2.65 s | 1.73 s |+| Language-model speed | 54.5 tokens/s | 97.2 tokens/s |## Benchmarks-Measured on an RTX 3060 (12 GB), Linux, Python 3.12 and NVIDIA driver 580.95.05,-using `weights="packed"`, the default decoder and the stable preset. These are-median end-to-end synthesis times from five runs after two warmups per voice.-Short, medium and long inputs contain 61, 226 and 680 characters; the long input-is split into two chunks. Model initialization and file writes are excluded.+One RTX 3060 (12 GB), driver 570.195.03 (CUDA 12.8), Ubuntu 24.04, Python 3.12, nothing else on the+GPU. PyTorch GPU is the default backend (PyTorch 2.14.1 CUDA 12.6 build, Transformers 5.20.2); the+vLLM columns use vLLM 0.31.0 (CUDA 12.9 build, PyTorch 2.13.0, Transformers 5.17.0) with default+`vllm_options`: "vLLM bf16" is without quantization, "vLLM 4-bit" adds `"quantization": "w4a16"`+([4-bit weights](#4-bit-weights)). Each time is the median of five `generate()` calls after two+warmups (`weights="packed"`, default decoder, stable preset, model loading excluded) on 61, 226 and+680 characters of text (the last in two chunks).-| Input / voice | PyTorch GPU | vLLM | Speedup |+| Input / voice | PyTorch GPU | vLLM bf16 | vLLM 4-bit | Speedup bf16 vs PyTorch | Speedup 4-bit vs PyTorch |+|---|---:|---:|---:|---:|---:|+| Short / Bruno | 4.44 s | 1.72 s | 1.18 s | 2.59x | 3.77x |+| Short / Bella | 6.14 s | 2.67 s | 1.62 s | 2.30x | 3.78x |+| Medium / Bruno | 14.74 s | 5.12 s | 3.43 s | 2.88x | 4.30x |+| Medium / Bella | 20.63 s | 7.01 s | 4.63 s | 2.94x | 4.46x |+| Long / Bruno | 45.20 s | 16.10 s | 10.38 s | 2.81x | 4.35x |+| Long / Bella | 53.15 s | 18.96 s | 12.24 s | 2.80x | 4.34x |++| RTX 3060 (12 GB) | PyTorch GPU | vLLM bf16 | vLLM 4-bit ||---|---:|---:|---:|-| Short / Bruno | 6.01 s | 2.05 s | 2.93x |-| Short / Bella | 6.98 s | 2.54 s | 2.75x |-| Medium / Bruno | 22.93 s | 5.49 s | 4.18x |-| Medium / Bella | 27.95 s | 8.26 s | 3.39x |-| Long / Bruno | 56.77 s | 17.56 s | 3.23x |-| Long / Bella | 79.24 s | 24.04 s | 3.30x |--The PyTorch column uses unchanged upstream code in a fresh normal installation-(PyTorch 2.14.1, Transformers 5.18.0). With versions matched to the vLLM install-(PyTorch 2.13.0, Transformers 5.17.0, vLLM 0.31.0), the measured speedup was-3.05-4.49x. Generated audio lengths differ between samplers; vLLM's synthesis-time was 0.44-0.48 seconds per second of output audio in these tests.--vLLM used about 8.5 GiB of GPU memory versus 5.8 GiB for the normal installation.-Cloning without `reference_text` also loads Whisper; those tests used up to-11.5 GiB. Supplying the transcript skips that model.-Initialization with cached downloads took 51 seconds versus 10 seconds; the-first vLLM startup took about 162 seconds. Reuse the loaded model across calls.+| Synthesis time per second of audio | 1.17-1.19 s | 0.41-0.45 s | 0.26-0.30 s |+| Peak GPU memory, whole card | 6,901 MiB | 9,026 MiB | 9,890 MiB |+| Start-up with files already downloaded (very first start) | 9.8 s (11.2 s) | 44.0 s (145.4 s) | 50.3 s (140.7 s) |++Per second of audio the speedups are 2.64-2.85x (bf16) and 3.94-4.51x (4-bit). Both vLLM columns+give vLLM the same share of the card, so 4-bit gets a larger cache (37,120 tokens instead of+18,512); its higher whole-card peak was in the main process, which decodes the audio, not in+vLLM's own process. Whole-card peaks include about 374 MiB the GPU reports as used when idle. A+full repeat of every column was within -0.9% to +3.4%.diff --git a/kittenml/get_model.py b/kittenml/get_model.pyindex b76afa1..284de9b 100644--- a/kittenml/get_model.py+++ b/kittenml/get_model.py@@ -71,8 +71,11 @@ class KittenTTS(KittenTTSOnnx):if model_type == KITTEN2_TYPE:from .kittentts2 import KittenTTS2if backend == "vllm":- from .kittentts2.vllm.backend import validate_environment+ from .kittentts2.vllm.backend import check_options, validate_environmentvalidate_environment(device)+ # Before the download: an unsupported GPU or a misspelled option+ # should not cost gigabytes.+ check_options(vllm_options, weights or config.get("default_weights", "packed"))elif vllm_options is not None:raise ValueError("vllm_options requires backend='vllm'")if local_dir is None:diff --git a/kittenml/kittentts2/model.py b/kittenml/kittentts2/model.pyindex 8d356ee..4a80b3e 100644--- a/kittenml/kittentts2/model.py+++ b/kittenml/kittentts2/model.py@@ -248,16 +248,18 @@ class KittenTTS2:from .weights import available_weightsreturn available_weights(self.config)- def _load_lm_weights(self, lm_dir):+ def _load_lm_weights(self, lm_dir, include=None):"""The language model's weights, in whichever packing was selected."packed" is a 1.58-bit packing that is exactly equivalent tomodel.safetensors at about a third of the size, and is the default."emb4" is smaller again but quantises the embedding, which is lossy β- see tl2_emb4.py. "full" reads the plain bf16 files.+ see tl2_emb4.py. "full" reads the plain bf16 files. `include`, a+ predicate on tensor names, limits loading to the tensors it accepts."""from .weights import load_lm_weights- return load_lm_weights(self.repo_dir, self.config, self.weights, lm_dir=lm_dir)+ return load_lm_weights(self.repo_dir, self.config, self.weights, lm_dir=lm_dir,+ include=include)def _load_voices(self):voices_path = self.repo_dir / self.config.get("voices", "voices/voices.json")diff --git a/kittenml/kittentts2/vllm/backend.py b/kittenml/kittentts2/vllm/backend.pyindex 25deebb..024c1ea 100644--- a/kittenml/kittentts2/vllm/backend.py+++ b/kittenml/kittentts2/vllm/backend.py@@ -10,16 +10,23 @@ import torch@contextlib.contextmanager-def _engine_environment():+def _engine_environment(forced=None):# KittenTTS already applies its warpers. Use vLLM's native sampler by default# so a pip installation needs no nvcc for FlashInfer's sampling JIT. The# worker inherits these settings, including quiet startup defaults. Restore- # the caller's environment afterwards and respect explicit overrides.+ # the caller's os.environ afterwards and respect explicit overrides.+ # `forced` values replace the caller's while the engine is created. vLLM reads+ # them when it builds the engine; in single-process mode+ # (VLLM_ENABLE_V1_MULTIPROCESSING=0) it then keeps its own frozen copy of the+ # environment for the rest of the process, which restoring os.environ does+ # not change (VLLMBackend drops that copy before creating each engine).defaults = {"VLLM_USE_FLASHINFER_SAMPLER": "0","VLLM_LOGGING_LEVEL": "WARNING", "TQDM_DISABLE": "1"}- previous = {key: os.environ.get(key) for key in defaults}+ forced = dict(forced or {})+ previous = {key: os.environ.get(key) for key in {**defaults, **forced}}for key, value in defaults.items():os.environ.setdefault(key, value)+ os.environ.update(forced)try:yieldfinally:@@ -40,6 +47,102 @@ def validate_environment(device=None):raise RuntimeError("the vLLM backend requires an NVIDIA CUDA GPU")if selected.index not in (None, 0):raise ValueError("select the GPU with CUDA_VISIBLE_DEVICES and use device='cuda'")+ major, minor = torch.cuda.get_device_capability()+ if major < 8:+ raise RuntimeError(f"this GPU (SM{major}{minor}) has no bfloat16 support, which "+ f"KittenTTS's language model needs: {_NO_FLOAT16}")+++# Engine settings KittenTTS chooses for vLLM, and their defaults.+_ENGINE_DEFAULTS = dict(gpu_memory_utilization=0.5, max_model_len=4096,+ max_num_seqs=8, enforce_eager=False)+# Options KittenTTS interprets itself. None keeps the default behaviour.+_KITTEN_OPTIONS = ("quantization", "dtype", "kv_cache_memory_bytes")+_QUANTIZATIONS = (None, "w4a16")+_DTYPES = {None: None, "bfloat16": torch.bfloat16}+# W4A16 runs on vLLM's Marlin kernel, the one whose exact output was verified.+# vLLM tries Machete first on Hopper (SM90) GPUs, so it is disabled for the+# engine; the loader then checks that every quantized layer uses Marlin.+_W4A16_KERNEL = "MarlinLinearKernel"+_W4A16_DISABLED_KERNELS = ("MacheteLinearKernel",)+# Measured with the current checkpoint, with and without W4A16: in float16 the+# language model's residual stream passes float16's maximum (65,504) at layer 10+# and overflows, after which it emits no audio tokens.+_NO_FLOAT16 = ("this model's residual stream exceeds the float16 range (measured overflow "+ "from layer 10); GPUs without bfloat16 (compute capability < 8.0) are not "+ "supported by the vLLM backend")+++def resolve_options(options):+ """Split `vllm_options` into engine keyword arguments and KittenTTS choices."""+ options = dict(options or {})+ unknown = set(options) - _ENGINE_DEFAULTS.keys() - set(_KITTEN_OPTIONS)+ if unknown:+ raise ValueError(f"unknown vllm_options: {sorted(unknown)}")+ kitten = {key: options.pop(key, None) for key in _KITTEN_OPTIONS}+ if kitten["quantization"] not in _QUANTIZATIONS:+ raise ValueError(f"vllm_options['quantization'] must be None or 'w4a16', "+ f"not {kitten['quantization']!r}")+ if kitten["dtype"] == "float16":+ raise ValueError(f"vllm_options['dtype']='float16' is not supported: {_NO_FLOAT16}")+ if kitten["dtype"] not in _DTYPES:+ raise ValueError(f"vllm_options['dtype'] must be None or 'bfloat16', "+ f"not {kitten['dtype']!r}")+ memory = kitten["kv_cache_memory_bytes"]+ if memory is not None and (isinstance(memory, bool) or not isinstance(memory, int)+ or memory <= 0):+ raise ValueError("vllm_options['kv_cache_memory_bytes'] must be a positive integer")+ engine = dict(_ENGINE_DEFAULTS)+ engine.update(options)+ if memory is not None:+ engine["kv_cache_memory_bytes"] = memory+ return engine, kitten+++def check_options(options, weights):+ """Validate `vllm_options` before anything is downloaded or loaded."""+ defaults, kitten = resolve_options(options)+ if kitten["quantization"] == "w4a16":+ check_w4a16_environment()+ return defaults, kitten+++def check_w4a16_environment(environ=None):+ """Refuse settings that would change what W4A16 computes or which kernel runs it."""+ environ = os.environ if environ is None else environ+ refuse = "vllm_options['quantization']='w4a16' refuses to start while "+ if "VLLM_MARLIN_INPUT_DTYPE" in environ:+ raise RuntimeError(refuse + "VLLM_MARLIN_INPUT_DTYPE is set: it quantizes activations "+ "to int8/fp8, turning W4A16 into W4A8. Unset it to run exact W4A16")+ if environ.get("VLLM_MARLIN_USE_ATOMIC_ADD") == "1": # vLLM reads only "1" as on+ raise RuntimeError(refuse + "VLLM_MARLIN_USE_ATOMIC_ADD=1: it would let Marlin reduce "+ "with unordered atomic adds. Unset it to run exact W4A16")+ disabled = environ.get("VLLM_DISABLED_KERNELS", "").split(",")+ if _W4A16_KERNEL in disabled:+ raise RuntimeError(refuse + f"VLLM_DISABLED_KERNELS names {_W4A16_KERNEL}, the only "+ "kernel exact W4A16 is verified with. Remove it from the list")+++def w4a16_engine_environment(environ=None):+ """VLLM_DISABLED_KERNELS for the engine: the caller's list plus Machete."""+ environ = os.environ if environ is None else environ+ disabled = [k for k in environ.get("VLLM_DISABLED_KERNELS", "").split(",") if k]+ disabled += [k for k in _W4A16_DISABLED_KERNELS if k not in disabled]+ return {"VLLM_DISABLED_KERNELS": ",".join(disabled)}+++def _main_process_tensor(name):+ """Only these LM tensors are used outside vLLM: prompt embeddings and spk_proj."""+ return name == "model.embed_tokens.weight" or name.startswith("spk_proj.")+++def quantization_args(quantization):+ """Engine arguments that select exact W4A16 (compressed-tensors, Marlin)."""+ if quantization is None:+ return {}+ from .w4a16 import quantization_config+ return {"quantization": "compressed-tensors",+ "hf_overrides": {"quantization_config": quantization_config()}}class VLLMBackend:@@ -48,40 +151,52 @@ class VLLMBackend:from transformers import AutoConfig, AutoTokenizerfrom ..speaker import load_speaker_head- options = dict(options or {})- defaults = dict(gpu_memory_utilization=0.5, max_model_len=4096,- max_num_seqs=8, enforce_eager=False)- unknown = set(options) - defaults.keys()- if unknown:- raise ValueError(f"unknown vllm_options: {sorted(unknown)}")- defaults.update(options)+ defaults, kitten = check_options(options, owner.weights)+ quantization = kitten["quantization"]+ forced_environment = w4a16_engine_environment() if quantization == "w4a16" else None+ default_dtype = (torch.bfloat16 if owner.config.get("dtype", "bf16") == "bf16"+ else torch.float32)+ dtype = _DTYPES[kitten["dtype"]] or default_dtype+ if quantization == "w4a16" and dtype is not torch.bfloat16:+ raise ValueError(f"W4A16 needs bfloat16 activations, not {dtype}")self.max_model_len = defaults["max_model_len"]self.token_map = owner.token_maplm_dir = owner.repo_dir / owner.config.get("lm_dir", "lm")config = AutoConfig.from_pretrained(str(lm_dir))if config.model_type != "qwen3":raise ValueError("the vLLM backend currently supports Qwen3 KittenTTS 2 checkpoints")- owner.dtype = torch.bfloat16 if owner.config.get("dtype", "bf16") == "bf16" else torch.float32+ owner.dtype = dtypeowner.tokenizer = AutoTokenizer.from_pretrained(str(lm_dir))- state = owner._load_lm_weights(lm_dir)+ # vLLM's own loader reads the full LM; this process only needs the+ # embedding and the speaker head, so do not expand the rest here.+ state = owner._load_lm_weights(lm_dir, include=_main_process_tensor)self.embedding = state["model.embed_tokens.weight"].to(owner.dtype)self.spk_proj = load_speaker_head(config.hidden_size, str(lm_dir),owner.device, owner.dtype,spk_dim=owner.config.get("speaker_embedding_dim", 512),state=state)del state- with _engine_environment():+ loader_config = {"repo_dir": str(owner.repo_dir), "config": owner.config,+ "weights": owner.weights}+ if quantization is not None:+ loader_config["quantization"] = quantization+ with _engine_environment(forced_environment):+ import vllm.envsfrom vllm import LLMfrom vllm.config import LoggingConfig+ # After an engine has started in this process (single-process mode),+ # vLLM answers from a frozen copy of the environment; drop it so this+ # engine sees the settings checked above.+ vllm.envs.disable_envs_cache()self.engine = LLM(model=str(lm_dir), tokenizer=str(lm_dir), dtype=owner.dtype,load_format="kittenml",- model_loader_extra_config={"repo_dir": str(owner.repo_dir),- "config": owner.config, "weights": owner.weights},+ model_loader_extra_config=loader_config,enable_prompt_embeds=True, enable_prefix_caching=False,logits_processors=["kittenml.kittentts2.vllm.logits:KittenLogitsProcessor"],- logging_config=LoggingConfig(), disable_log_stats=True, **defaults)+ logging_config=LoggingConfig(), disable_log_stats=True,+ **quantization_args(quantization), **defaults)def prepare_request(self, prompt, speaker_embedding, settings, max_new_tokens, advanced):from vllm import SamplingParamsdiff --git a/kittenml/kittentts2/vllm/loader.py b/kittenml/kittentts2/vllm/loader.pyindex 96b01f7..3d0fd06 100644--- a/kittenml/kittentts2/vllm/loader.py+++ b/kittenml/kittentts2/vllm/loader.py@@ -12,14 +12,90 @@ class KittenModelLoader(BaseModelLoader):def load_weights(self, model, model_config):options = self.load_config.model_loader_extra_config- state = load_lm_weights(options["repo_dir"], options["config"], options["weights"])- loaded = model.load_weights((name, tensor) for name, tensor in state.items()- if not name.startswith("spk_proj."))- missing = set(dict(model.named_parameters())) - loaded+ quantization = options.get("quantization")+ if quantization == "w4a16":+ # vLLM chose each layer's kernel when it built the model; check before loading.+ _check_w4a16_kernels(model, model_config)+ # Exact 4-bit weights straight from the trits, one tensor at a time.+ from .w4a16 import iter_w4a16+ weights = iter_w4a16(options["repo_dir"], options["config"], options["weights"])+ elif quantization is None:+ state = load_lm_weights(options["repo_dir"], options["config"], options["weights"])+ weights = ((name, tensor) for name, tensor in state.items()+ if not name.startswith("spk_proj."))+ else:+ raise ValueError(f"unknown KittenTTS quantization {quantization!r}")+ loaded = model.load_weights(weights)+ missing = set(dict(model.named_parameters())) - loaded - _kv_cache_scale_names(model)if missing:raise ValueError(f"weights missing from the checkpoint: {sorted(missing)[:5]}")+def _quantized_layers(model):+ """(name, module, kernel) for every linear layer vLLM built with a mixed-precision kernel."""+ for name, module in model.named_modules():+ kernel = getattr(getattr(module, "scheme", None), "kernel", None)+ if kernel is not None:+ yield name, module, kernel+++def _check_w4a16_kernels(model, model_config):+ """Every quantized linear layer must run exact W4A16 on Marlin.++ vLLM picks the mixed-precision kernel per GPU (Machete on Hopper, Marlin+ elsewhere, others when a kernel is disabled). Exact W4A16 was verified with+ Marlin only, so refuse to load if vLLM chose anything else. This runs in the+ engine process: VLLM_MARLIN_INPUT_DTYPE keeps the Marlin kernel but quantizes+ its activations to 8 bits (W4A8), which shows in the kernel's activation type+ (in vLLM 0.31.0 that is the only way the setting acts).+ """+ import torch+ from vllm.model_executor.kernels.linear import MarlinLinearKernel++ refuse = "W4A16: refusing to load: "+ if model_config.dtype != torch.bfloat16:+ raise RuntimeError(refuse + f"the engine dtype is {model_config.dtype}, not bfloat16")+ chosen, activations = {}, {}+ for name, _, kernel in _quantized_layers(model):+ chosen.setdefault(type(kernel).__name__, []).append(name)+ act_type = getattr(getattr(kernel, "config", None), "act_type", None)+ activations.setdefault(str(act_type), []).append(name)+ if not chosen:+ raise RuntimeError("W4A16: vLLM created no quantized linear layers")+ other = {kind: names for kind, names in chosen.items()+ if kind != MarlinLinearKernel.__name__}+ if other:+ layers = sum(map(len, other.values()))+ raise RuntimeError(f"W4A16: vLLM chose {sorted(other)} for {layers} linear layers, but "+ "exact W4A16 is verified only with MarlinLinearKernel (check "+ "VLLM_DISABLED_KERNELS; with VLLM_ENABLE_V1_MULTIPROCESSING=0 vLLM "+ "may still use the value from the first engine in this process)")+ wrong = {act: names for act, names in activations.items() if act != str(model_config.dtype)}+ if wrong:+ layers = sum(map(len, wrong.values()))+ raise RuntimeError(refuse + f"{layers} linear layers would quantize their activations to "+ f"{sorted(wrong)} instead of computing in {model_config.dtype} (check "+ "VLLM_MARLIN_INPUT_DTYPE)")+++def _kv_cache_scale_names(model):+ """Attention KV-cache scales a quantization method adds, which the checkpoint omits.++ With quantization="compressed-tensors", every attention layer gets q/k/v scale+ and zero-point parameters for a quantized KV cache. KittenTTS keeps the KV cache+ in the model dtype, so they keep vLLM's defaults (as with vLLM's own loader);+ every other parameter must still come from the checkpoint.+ """+ from vllm.model_executor.layers.quantization.kv_cache import BaseKVCacheMethod++ names = set()+ for prefix, module in model.named_modules():+ if isinstance(getattr(module, "quant_method", None), BaseKVCacheMethod):+ names.update(f"{prefix}.{name}" if prefix else name+ for name, _ in module.named_parameters(recurse=False))+ return names++def register():from vllm.model_executor.model_loader import register_model_loaderregister_model_loader("kittenml")(KittenModelLoader)diff --git a/kittenml/kittentts2/vllm/w4a16.py b/kittenml/kittentts2/vllm/w4a16.pynew file mode 100644index 0000000..117e9d9--- /dev/null+++ b/kittenml/kittentts2/vllm/w4a16.py@@ -0,0 +1,227 @@+"""Exact 4-bit (W4A16) weights for the vLLM backend.++The language model's 196 linear projections are ternary: every 128-wide input+group of every output row holds only {-s, 0, +s}, with one bf16 scale per group+(see ternary.py). That is exactly what a symmetric 4-bit, group-128 weight can+represent: compressed-tensors' ``uint4b8`` type stores ``q = sign + 8``, so the+codes are 7, 8 and 9, and dequantising gives ``(q - 8) * s``, the original bf16+weight bit for bit. The group scales are copied through unchanged.++vLLM then runs these layers with its Marlin mixed-precision kernel: 4-bit+weights, bf16 activations (the backend and loader keep vLLM from choosing+another kernel or 8-bit activations). Everything that is not ternary stays at+its stored precision: the token embedding and the tied output head, all norms,+and the KV cache. ``spk_proj`` runs outside vLLM and is dropped here.++The lossy ``emb4`` packing is refused: W4A16 is offered as an exact+representation of the published weights and is never combined with it.++vLLM fuses q/k/v into ``qkv_proj`` and gate/up into ``gate_up_proj``+(``Qwen3ForCausalLM.packed_modules_mapping``), so the 196 source projections+(7 per layer) become 4 quantized modules per layer: 112 for the 28-layer model.++This module does not import vLLM, so it can be tested on any machine.+"""++import copy+import json+import re++import torch++from .. import ternary+from ..weights import plain_shards, weights_path++NUM_BITS = 4+GROUP_SIZE = ternary.GROUP_SIZE # 128, as in the checkpoint+OFFSET = 1 << (NUM_BITS - 1) # 8: uint4b8 stores sign + 8+PACK_FACTOR = 32 // NUM_BITS # 8 codes per int32++# The compressed-tensors config vLLM reads from hf_config.quantization_config.+# Targets match vLLM's linear classes by name (QKVParallelLinear,+# MergedColumnParallelLinear, RowParallelLinear); the vocabulary embedding is+# not a Linear and stays unquantized, and the tied head is ignored explicitly.+QUANTIZATION_CONFIG = {+ "quant_method": "compressed-tensors",+ "format": "pack-quantized",+ "config_groups": {+ "group_0": {+ "targets": ["Linear"],+ "weights": {+ "num_bits": NUM_BITS,+ "type": "int",+ "symmetric": True,+ "strategy": "group",+ "group_size": GROUP_SIZE,+ "dynamic": False,+ "actorder": None,+ },+ "input_activations": None,+ },+ },+ "ignore": ["lm_head"],+}++# The checkpoint's ternary projections, by name. Anything else passes through.+_PROJECTION = re.compile(+ r"^model\.layers\.(\d+)\.(self_attn\.(?:q|k|v|o)_proj|mlp\.(?:gate|up|down)_proj)\.weight$")++def quantization_config():+ """A fresh copy: vLLM adds keys (head counts) to the dict it is given."""+ return copy.deepcopy(QUANTIZATION_CONFIG)+++def is_projection(name):+ """True for the q/k/v/o/gate/up/down weights that become 4-bit."""+ return _PROJECTION.match(name) is not None+++def pack_int4(signs):+ """int8 values in [-8, 7] -> int32 words, compressed-tensors' bit order.++ Matches ``compressed_tensors.compressors.pack_quantized.helpers.pack_to_int32``+ with ``num_bits=4, packed_dim=1``: each value is shifted to unsigned+ (+8), and column ``8*j + i`` of a row occupies bits ``4*i`` to ``4*i + 3`` of+ word ``j``, lowest column in the lowest bits.+ """+ if signs.dtype != torch.int8 or signs.dim() != 2:+ raise ValueError("expected a 2-D int8 tensor")+ rows, cols = signs.shape+ if cols % PACK_FACTOR:+ raise ValueError(f"input width {cols} is not a multiple of {PACK_FACTOR}")+ codes = (signs.to(torch.int64) + OFFSET).reshape(rows, cols // PACK_FACTOR, PACK_FACTOR)+ if codes.min() < 0 or codes.max() >= 1 << NUM_BITS:+ raise ValueError("values do not fit in 4 bits")+ shifts = torch.arange(PACK_FACTOR, dtype=torch.int64) * NUM_BITS+ words = (codes << shifts).sum(dim=2)+ # Reinterpret the unsigned 32-bit words as int32 without relying on overflow.+ words = torch.where(words >= 1 << 31, words - (1 << 32), words)+ return words.to(torch.int32).contiguous()+++def bit_equal(a, b):+ """Same dtype, shape and bit pattern. Unlike torch.equal, -0.0 differs from +0.0."""+ if a.dtype != b.dtype or a.shape != b.shape:+ return False+ if not a.is_floating_point():+ return torch.equal(a, b)+ bits = {1: torch.uint8, 2: torch.int16, 4: torch.int32, 8: torch.int64}[a.element_size()]+ return torch.equal(a.contiguous().view(bits), b.contiguous().view(bits))+++def _quantized(name, signs, scale):+ """Yield the three compressed-tensors parameters for one projection."""+ rows, cols = signs.shape+ if scale.shape != (rows, cols // GROUP_SIZE):+ raise ValueError(f"{name}: scale shape {tuple(scale.shape)} does not match "+ f"[{rows}, {cols // GROUP_SIZE}]")+ if scale.dtype != torch.bfloat16:+ raise ValueError(f"{name}: group scales are {scale.dtype}, not bfloat16")+ base = name.removesuffix(".weight")+ yield f"{base}.weight_packed", pack_int4(signs)+ yield f"{base}.weight_scale", scale.contiguous()+ yield f"{base}.weight_shape", torch.tensor([rows, cols], dtype=torch.int64)+++def _from_dense(name, tensor):+ """A dense projection that must be exactly ternary, as 4-bit parameters."""+ if tensor.dim() != 2 or tensor.shape[1] % GROUP_SIZE:+ raise ValueError(f"{name}: shape {tuple(tensor.shape)} cannot be grouped by {GROUP_SIZE}")+ trits, scale = ternary._to_trits(tensor, GROUP_SIZE)+ rebuilt = ternary._from_trits(trits, scale, tensor.shape, GROUP_SIZE, tensor.dtype)+ if not bit_equal(rebuilt, tensor):+ raise ValueError(f"{name} is not exactly ternary in {GROUP_SIZE}-wide groups; "+ "refusing to quantize it to W4A16, which would be lossy")+ signs = (trits.to(torch.int8) - 1).reshape(tensor.shape)+ yield from _quantized(name, signs, scale)+++def _iter_packed(path):+ from safetensors import safe_open++ with safe_open(str(path), framework="pt") as handle:+ metadata = handle.metadata() or {}+ if metadata.get("format") != "kittenml-ternary-1.58":+ raise ValueError(f"not a packed ternary checkpoint: {metadata.get('format')!r}")+ if int(metadata["group_size"]) != GROUP_SIZE:+ raise ValueError(f"packed group size {metadata['group_size']} is not {GROUP_SIZE}")+ packed = json.loads(metadata["packed"])+ for name in handle.keys():+ if "::" in name or name.startswith("spk_proj."):+ continue+ tensor = handle.get_tensor(name)+ if is_projection(name):+ yield from _from_dense(name, tensor)+ else:+ yield name, tensor+ for name, spec in packed.items():+ if name.startswith("spk_proj."):+ continue+ shape = torch.Size(spec["shape"])+ trits = ternary._unpack_trits(handle.get_tensor(f"{name}::codes"), shape.numel())+ scale = handle.get_tensor(f"{name}::scale")+ if not is_projection(name):+ # Not a layer vLLM quantizes: hand it over dense, exactly as unpacked.+ yield name, ternary._from_trits(trits, scale, shape, GROUP_SIZE,+ getattr(torch, spec["dtype"]))+ continue+ signs = (trits.to(torch.int8) - 1).reshape(shape)+ yield from _quantized(name, signs, scale)+++def _iter_emb4(repo_dir, config):+ from ..weights import load_lm_weights++ state = load_lm_weights(repo_dir, config, "emb4")+ for name, tensor in state.items():+ if name.startswith("spk_proj."):+ continue+ if is_projection(name):+ yield from _from_dense(name, tensor)+ else:+ yield name, tensor+++def _iter_full(lm_dir, shards):+ from safetensors import safe_open++ if not shards:+ raise FileNotFoundError(f"no plain model weights in {lm_dir}")+ for shard in shards:+ with safe_open(str(shard), framework="pt") as handle:+ for name in handle.keys():+ if name.startswith("spk_proj."):+ continue+ tensor = handle.get_tensor(name)+ if is_projection(name):+ yield from _from_dense(name, tensor)+ else:+ yield name, tensor+++def iter_w4a16(repo_dir, config, variant):+ """Yield (name, tensor) pairs for vLLM's compressed-tensors W4A16 layers.++ One tensor at a time, so the expanded bf16 state dict is never built. (The+ bf16 files are memory-mapped, so with ``variant="full"`` the pages read so+ far still count towards the process's resident memory.)++ - ``variant="packed"`` reads the 1.58-bit file and converts trits directly.+ - ``variant="full"`` reads the bf16 files and checks every projection is+ exactly ternary, raising otherwise.+ - ``variant="emb4"`` unpacks the TL2 body (exactly ternary, checked like+ ``"full"``) and keeps its 4-bit embedding, which is lossy; only the+ embedding and the tied output head differ from the bf16 model.++ Projections become ``weight_packed`` (int32), ``weight_scale`` (the stored+ bf16 scales, unchanged) and ``weight_shape`` (int64 [2]). Other tensors pass+ through; ``spk_proj.*`` is dropped because it runs outside vLLM.+ """+ if variant == "emb4":+ return _iter_emb4(repo_dir, config)+ path = weights_path(repo_dir, config, variant)+ if path is None:+ return _iter_full(*plain_shards(repo_dir, config))+ if not path.is_file():+ raise FileNotFoundError(f"{variant!r} weights not found: {path}")+ return _iter_packed(path)diff --git a/kittenml/kittentts2/weights.py b/kittenml/kittentts2/weights.pyindex cc52c39..77f2242 100644--- a/kittenml/kittentts2/weights.py+++ b/kittenml/kittentts2/weights.py@@ -1,5 +1,6 @@"""Read the selected checkpoint identically for Torch and vLLM."""+import jsonfrom pathlib import Path@@ -20,8 +21,20 @@ def weights_path(repo_dir, config, name):return Path(repo_dir) / declared-def load_lm_weights(repo_dir, config, variant, lm_dir=None):- """Unpack on CPU without re-quantizing or writing an expanded checkpoint."""+def plain_shards(repo_dir, config, lm_dir=None):+ """The bf16 ("full") safetensors files of the language model, in load order."""+ lm_dir = Path(lm_dir) if lm_dir is not None else Path(repo_dir) / config.get("lm_dir", "lm")+ return lm_dir, [s for s in sorted(lm_dir.glob("model*.safetensors")) if "-" not in s.stem]+++def load_lm_weights(repo_dir, config, variant, lm_dir=None, include=None):+ """Unpack on CPU without re-quantizing or writing an expanded checkpoint.++ `include`, if given, is a predicate on tensor names: only those tensors are+ read (and, for packed files, unpacked), so callers that need a few tensors+ do not expand the whole model in host memory. The emb4 reader always+ decodes the full file and filters afterwards.+ """from safetensors import safe_openfrom safetensors.torch import load_file@@ -31,18 +44,30 @@ def load_lm_weights(repo_dir, config, variant, lm_dir=None):raise FileNotFoundError(f"{variant!r} weights not found: {path}")if variant == "emb4":from .tl2_emb4 import load_state_dict- return load_state_dict(path)+ state = load_state_dict(path)+ return state if include is None else {k: v for k, v in state.items() if include(k)}with safe_open(str(path), framework="pt") as handle:metadata = handle.metadata() or {}- tensors = {k: handle.get_tensor(k) for k in handle.keys()}+ if include is None:+ tensors = {k: handle.get_tensor(k) for k in handle.keys()}+ else:+ tensors = {k: handle.get_tensor(k) for k in handle.keys()+ if include(k.split("::", 1)[0])}+ if include is not None and "packed" in metadata:+ packed = json.loads(metadata["packed"])+ metadata = {**metadata, "packed": json.dumps(+ {k: v for k, v in packed.items() if include(k)})}from .ternary import unpack_state_dictreturn unpack_state_dict(tensors, metadata)- lm_dir = Path(lm_dir) if lm_dir is not None else Path(repo_dir) / config.get("lm_dir", "lm")+ lm_dir, shards = plain_shards(repo_dir, config, lm_dir)state = {}- for shard in sorted(lm_dir.glob("model*.safetensors")):- if "-" not in shard.stem:+ for shard in shards:+ if include is None:state.update(load_file(str(shard)))- if not state:+ continue+ with safe_open(str(shard), framework="pt") as handle:+ state.update({k: handle.get_tensor(k) for k in handle.keys() if include(k)})+ if not shards or (include is None and not state):raise FileNotFoundError(f"no plain model weights in {lm_dir}")return statediff --git a/tests/test_vllm.py b/tests/test_vllm.pyindex 2a16ebd..f0744e2 100644--- a/tests/test_vllm.py+++ b/tests/test_vllm.py@@ -199,6 +199,7 @@ def test_engine_receives_quiet_logging_config_without_changing_native_options(moreturn mock.sentinel.enginewith mock.patch("kittenml.kittentts2.vllm.backend.validate_environment"), \+ mock.patch("torch.cuda.get_device_capability", return_value=(8, 6)), \mock.patch("transformers.AutoConfig.from_pretrained", return_value=SimpleNamespace(model_type="qwen3", hidden_size=4)), \mock.patch("transformers.AutoTokenizer.from_pretrained"), \@@ -224,6 +225,9 @@ def test_vllm_loader_filters_speaker_weights_and_rejects_missing_parameters():def named_parameters(self):return [("model.weight", None)]+ def named_modules(self):+ return []+model = TinyModel()state = {"model.weight": torch.ones(2), "spk_proj.0.bias": torch.zeros(2)}with mock.patch("kittenml.kittentts2.vllm.loader.load_lm_weights", return_value=state):@@ -292,3 +296,564 @@ def test_sampler_is_neutral_and_speaker_replaces_only_first_embedding():assert params.extra_args["kittenml"]["sampling"]["temperature"] == 0.7with pytest.raises(ValueError, match="token budget"):backend.prepare_request([0, 3, 1], torch.ones(1, 4), settings, 30, advanced)+++# ββ Exact W4A16 (vllm_options={"quantization": "w4a16"}) βββββββββββββββββββββββ++_PROJECTIONS = {"self_attn.q_proj": (256, 256), "self_attn.k_proj": (128, 256),+ "self_attn.v_proj": (128, 256), "self_attn.o_proj": (256, 256),+ "mlp.gate_proj": (384, 256), "mlp.up_proj": (384, 256),+ "mlp.down_proj": (256, 384)}+++def _ternary(rows, cols, generator):+ signs = torch.randint(-1, 2, (rows, cols), generator=generator)+ scale = (torch.rand(rows, cols // 128, generator=generator) * 0.5 + 1e-4).to(torch.bfloat16)+ scale[0, 0] = 0 # an all-zero group has a zero scale+ signs[0, :128] = 0+ return signs.to(torch.bfloat16) * scale.repeat_interleave(128, dim=1)+++def _w4a16_checkpoint(tmp_path, layers=2, perturb=None):+ """A small Qwen3-shaped LM written as both full bf16 and packed ternary."""+ generator = torch.Generator().manual_seed(0)+ state = {"model.embed_tokens.weight": torch.randn(32, 256, generator=generator).to(torch.bfloat16),+ "model.norm.weight": torch.rand(256, generator=generator).to(torch.bfloat16),+ "spk_proj.0.weight": torch.randn(256, 8, generator=generator).to(torch.bfloat16),+ "spk_proj.0.bias": torch.zeros(256, dtype=torch.bfloat16),+ "spk_proj.1.weight": torch.ones(256, dtype=torch.bfloat16),+ "spk_proj.1.bias": torch.zeros(256, dtype=torch.bfloat16)}+ for layer in range(layers):+ prefix = f"model.layers.{layer}."+ state[prefix + "input_layernorm.weight"] = torch.ones(256, dtype=torch.bfloat16)+ state[prefix + "self_attn.q_norm.weight"] = torch.ones(128, dtype=torch.bfloat16)+ for name, shape in _PROJECTIONS.items():+ state[prefix + name + ".weight"] = _ternary(*shape, generator)+ if perturb is not None:+ weight = state[perturb]+ weight[1, 5] = weight[1, :128].abs().max() / 2 # no longer ternary in that group+ lm = tmp_path / "lm"+ lm.mkdir()+ save_file(state, lm / "model.safetensors")+ tensors, metadata = pack_state_dict(state)+ save_file(tensors, lm / "model-ternary.safetensors", metadata=metadata)+ return state, {"lm_packed": "lm/model-ternary.safetensors"}+++def _unpack_int4(packed, shape):+ """Inverse of w4a16.pack_int4: int32 words -> int8 values of `shape`."""+ from kittenml.kittentts2.vllm.w4a16 import NUM_BITS, OFFSET, PACK_FACTOR++ words = packed.to(torch.int64) & 0xFFFFFFFF+ shifts = torch.arange(PACK_FACTOR, dtype=torch.int64) * NUM_BITS+ codes = (words.unsqueeze(2) >> shifts) & ((1 << NUM_BITS) - 1)+ return (codes - OFFSET).reshape(packed.shape[0], -1)[:, :int(shape[1])].to(torch.int8)+++def _reference_unpack():+ """compressed-tensors' own unpacker when installed (it ships with vLLM)."""+ try:+ from compressed_tensors.compressors.pack_quantized.helpers import unpack_from_int32+ except ImportError:+ return _unpack_int4+ return lambda packed, shape: unpack_from_int32(packed, 4, torch.Size(shape))++[email protected]("variant", ["packed", "full"])+def test_w4a16_dequantizes_bit_exactly_to_the_selected_weights(tmp_path, variant):+ from kittenml.kittentts2.vllm.w4a16 import bit_equal, is_projection, iter_w4a16++ state, config = _w4a16_checkpoint(tmp_path)+ expected = load_lm_weights(tmp_path, config, variant)+ produced = dict(iter_w4a16(tmp_path, config, variant))+ unpack = _reference_unpack()+ projections = [name for name in state if is_projection(name)]+ assert len(projections) == 14+ dense = {name for name in state if not is_projection(name) and not name.startswith("spk_proj.")}+ assert set(produced) == dense | {name.removesuffix(".weight") + suffix+ for name in projections+ for suffix in (".weight_packed", ".weight_scale",+ ".weight_shape")}+ for name in dense:+ assert bit_equal(produced[name], expected[name])+ for name in projections:+ base = name.removesuffix(".weight")+ rows, cols = expected[name].shape+ packed = produced[base + ".weight_packed"]+ scale = produced[base + ".weight_scale"]+ shape = produced[base + ".weight_shape"]+ assert packed.dtype == torch.int32 and packed.shape == (rows, cols // 8)+ assert scale.dtype == torch.bfloat16 and scale.shape == (rows, cols // 128)+ assert shape.dtype == torch.int64 and shape.tolist() == [rows, cols]+ signs = unpack(packed, shape.tolist())+ assert set((signs.to(torch.int16) + 8).unique().tolist()) <= {7, 8, 9}+ weight = signs.to(torch.bfloat16) * scale.repeat_interleave(128, dim=1)+ assert bit_equal(weight, expected[name]), name+++def test_w4a16_packed_and_full_sources_give_identical_tensors(tmp_path):+ from kittenml.kittentts2.vllm.w4a16 import bit_equal, iter_w4a16++ _, config = _w4a16_checkpoint(tmp_path)+ packed = dict(iter_w4a16(tmp_path, config, "packed"))+ full = dict(iter_w4a16(tmp_path, config, "full"))+ assert packed.keys() == full.keys()+ assert all(bit_equal(packed[k], full[k]) for k in packed)+++def test_w4a16_int32_bit_order_matches_compressed_tensors():+ from kittenml.kittentts2.vllm.w4a16 import pack_int4++ # Codes 7,8,9,8,8,8,8,9: column 0 in bits 0-3 ... column 7 in bits 28-31.+ golden = pack_int4(torch.tensor([[-1, 0, 1, 0, 0, 0, 0, 1]], dtype=torch.int8))+ assert golden.dtype == torch.int32 and golden.tolist() == [[-1735882361]] # 0x98888987+ assert pack_int4(torch.zeros(1, 8, dtype=torch.int8)).tolist() == [[0x88888888 - (1 << 32)]]+ values = torch.randint(-8, 8, (6, 256), generator=torch.Generator().manual_seed(1)).to(torch.int8)+ assert torch.equal(_unpack_int4(pack_int4(values), values.shape), values)+ with pytest.raises(ValueError, match="4 bits"):+ pack_int4(torch.full((1, 8), 8, dtype=torch.int8))+ helpers = pytest.importorskip("compressed_tensors.compressors.pack_quantized.helpers")+ assert torch.equal(pack_int4(values), helpers.pack_to_int32(values, 4))+ assert torch.equal(helpers.unpack_from_int32(pack_int4(values), 4, values.shape), values)++[email protected]("variant", ["packed", "full"])+def test_w4a16_refuses_weights_that_are_not_exactly_ternary(tmp_path, variant):+ from kittenml.kittentts2.vllm.w4a16 import iter_w4a16++ _, config = _w4a16_checkpoint(tmp_path, perturb="model.layers.1.mlp.up_proj.weight")+ with pytest.raises(ValueError, match="model.layers.1.mlp.up_proj.weight is not exactly ternary"):+ list(iter_w4a16(tmp_path, config, variant))+++def test_w4a16_refuses_scales_that_are_not_bfloat16(tmp_path):+ # Marlin reads the group scales in the activation dtype, which is bfloat16.+ from kittenml.kittentts2.vllm.w4a16 import iter_w4a16++ weight = torch.zeros(2, 128, dtype=torch.float32)+ weight[0, :2] = torch.tensor([0.5, -0.5])+ (tmp_path / "lm").mkdir()+ save_file({"model.layers.0.self_attn.o_proj.weight": weight}, tmp_path / "lm" / "model.safetensors")+ with pytest.raises(ValueError, match="o_proj.weight: group scales are torch.float32, not bfloat16"):+ list(iter_w4a16(tmp_path, {}, "full"))+++def test_w4a16_accepts_emb4_and_packs_its_ternary_body_exactly(tmp_path):+ from kittenml.kittentts2.vllm import w4a16+ from kittenml.kittentts2.vllm.backend import check_options++ signs = torch.randint(-1, 2, (16, 256), dtype=torch.int8)+ scale = torch.rand(16, 2, dtype=torch.bfloat16) + 0.5+ weight = (signs.reshape(16, 2, 128).to(torch.float32)+ * scale.to(torch.float32).unsqueeze(-1)).reshape(16, 256).to(torch.bfloat16)+ embedding = torch.randn(8, 4, dtype=torch.bfloat16)+ state = {"model.layers.0.self_attn.o_proj.weight": weight,+ "model.embed_tokens.weight": embedding,+ "spk_proj.weight": torch.zeros(2, 2, dtype=torch.bfloat16)}+ with mock.patch("kittenml.kittentts2.weights.load_lm_weights", return_value=state) as load:+ out = dict(w4a16.iter_w4a16(tmp_path, {"lm_emb4": "x"}, "emb4"))+ load.assert_called_once_with(tmp_path, {"lm_emb4": "x"}, "emb4")+ assert "spk_proj.weight" not in out+ assert torch.equal(out["model.embed_tokens.weight"], embedding)+ expected = dict(w4a16._from_dense("model.layers.0.self_attn.o_proj.weight", weight))+ for key, value in expected.items():+ assert torch.equal(out[key], value)++ check_options({"quantization": "w4a16"}, "emb4")+++def test_w4a16_targets_112_fused_modules_for_196_projections():+ # Qwen3-1.7B shape: 28 layers x 7 ternary projections. vLLM's own Qwen3 model+ # loads q/k/v into qkv_proj and gate/up into gate_up_proj (its+ # packed_modules_mapping), so 4 quantized modules per layer.+ pytest.importorskip("vllm")+ from vllm.model_executor.models.qwen3 import Qwen3ForCausalLM+ from kittenml.kittentts2.vllm.w4a16 import is_projection++ fused = {part: module for module, parts in Qwen3ForCausalLM.packed_modules_mapping.items()+ for part in parts}+ names = ["model.embed_tokens.weight", "model.norm.weight", "spk_proj.0.weight"]+ for layer in range(28):+ names += [f"model.layers.{layer}.{p}.weight" for p in _PROJECTIONS]+ names += [f"model.layers.{layer}.input_layernorm.weight",+ f"model.layers.{layer}.self_attn.k_norm.weight"]+ projections = [name.removesuffix(".weight") for name in names if is_projection(name)]+ assert len(projections) == 196+ modules = {name.rsplit(".", 1)[0] + "." + fused.get(name.rsplit(".", 1)[1], name.rsplit(".", 1)[1])+ for name in projections}+ assert len(modules) == 112+ assert {m.rsplit(".", 1)[1] for m in modules} == {"qkv_proj", "o_proj", "gate_up_proj",+ "down_proj"}+++def test_vllm_options_validate_quantization_dtype_and_cache_size():+ from kittenml.kittentts2.vllm.backend import resolve_options++ engine, kitten = resolve_options(None)+ assert engine == dict(gpu_memory_utilization=0.5, max_model_len=4096, max_num_seqs=8,+ enforce_eager=False)+ assert kitten == dict(quantization=None, dtype=None, kv_cache_memory_bytes=None)+ engine, kitten = resolve_options({"quantization": "w4a16", "dtype": "bfloat16",+ "kv_cache_memory_bytes": 1 << 30, "max_num_seqs": 4})+ assert engine["kv_cache_memory_bytes"] == 1 << 30 and engine["max_num_seqs"] == 4+ assert "quantization" not in engine and "dtype" not in engine+ assert kitten["quantization"] == "w4a16" and kitten["dtype"] == "bfloat16"+ for bad, match in [({"quantization": "w8a8"}, "quantization"),+ ({"quantization": "W4A16"}, "quantization"),+ ({"dtype": "fp16"}, "dtype"),+ ({"dtype": "float32"}, "dtype"),+ ({"kv_cache_memory_bytes": 0}, "kv_cache_memory_bytes"),+ ({"kv_cache_memory_bytes": 1.5e9}, "kv_cache_memory_bytes"),+ ({"kv_cache_memory_bytes": True}, "kv_cache_memory_bytes"),+ ({"kv_cache_dtype": "fp8"}, "unknown vllm_options")]:+ with pytest.raises(ValueError, match=match):+ resolve_options(bad)+++def test_float16_is_refused_with_the_measured_reason():+ # float16 was measured to overflow at layer 10 with the current checkpoint+ # (with and without W4A16) and then produce no audio, so it is refused.+ from kittenml.kittentts2.vllm.backend import resolve_options++ for options in ({"dtype": "float16"}, {"dtype": "float16", "quantization": "w4a16"}):+ with pytest.raises(ValueError, match=r"'float16' is not supported: this model's residual "+ r"stream exceeds the float16 range \(measured overflow "+ r"from layer 10\).*compute capability < 8.0"):+ resolve_options(options)+++def test_dtype_follows_hardware_and_gpus_without_bfloat16_are_refused():+ from kittenml.kittentts2.vllm.backend import validate_environment++ with mock.patch("kittenml.kittentts2.vllm.backend.importlib.util.find_spec", return_value=object()), \+ mock.patch("torch.cuda.is_available", return_value=True):+ for capability in ((8, 0), (8, 6), (12, 0)):+ with mock.patch("torch.cuda.get_device_capability", return_value=capability):+ validate_environment()+ for capability, name in (((7, 5), "SM75"), ((7, 0), "SM70")):+ with mock.patch("torch.cuda.get_device_capability", return_value=capability):+ with pytest.raises(RuntimeError, match=name + ".*no bfloat16.*float16 range"+ ".*layer 10.*< 8.0"):+ validate_environment()+++def test_vllm_options_are_checked_before_model_download(monkeypatch):+ resolved = (None, "example/repo", {"type": "KITTEN2"})+ with mock.patch("kittenml.get_model.resolve_repo", return_value=resolved), \+ mock.patch("kittenml.kittentts2.vllm.backend.validate_environment"), \+ mock.patch("kittenml.get_model.snapshot_download") as download:+ for options, weights, match in [({"quantisation": "w4a16"}, None, "unknown vllm_options"),+ ({"dtype": "float16"}, None, "float16 range")]:+ with pytest.raises(ValueError, match=match):+ KittenTTS("example/repo", backend="vllm", weights=weights, vllm_options=options)+ monkeypatch.setenv("VLLM_MARLIN_INPUT_DTYPE", "int8")+ with pytest.raises(RuntimeError, match="W4A8"):+ KittenTTS("example/repo", backend="vllm", vllm_options={"quantization": "w4a16"})+ download.assert_not_called()+++def test_gpus_without_bfloat16_are_refused_before_model_download():+ resolved = (None, "example/repo", {"type": "KITTEN2"})+ with mock.patch("kittenml.get_model.resolve_repo", return_value=resolved), \+ mock.patch("kittenml.kittentts2.vllm.backend.importlib.util.find_spec", return_value=object()), \+ mock.patch("torch.cuda.is_available", return_value=True), \+ mock.patch("torch.cuda.get_device_capability", return_value=(7, 5)), \+ mock.patch("kittenml.get_model.snapshot_download") as download:+ with pytest.raises(RuntimeError, match="SM75.*no bfloat16"):+ KittenTTS("example/repo", backend="vllm")+ download.assert_not_called()+++def test_w4a16_refuses_marlin_environment_overrides(monkeypatch, tmp_path):+ from kittenml.kittentts2.vllm.backend import VLLMBackend, check_w4a16_environment++ check_w4a16_environment({})+ # vLLM reads only VLLM_MARLIN_USE_ATOMIC_ADD=1 as on; other kernels may be disabled.+ check_w4a16_environment({"VLLM_MARLIN_USE_ATOMIC_ADD": "0",+ "VLLM_DISABLED_KERNELS": "ExllamaLinearKernel"})+ for key, value in (("VLLM_MARLIN_INPUT_DTYPE", "int8"), ("VLLM_MARLIN_INPUT_DTYPE", "fp8"),+ ("VLLM_MARLIN_USE_ATOMIC_ADD", "1"),+ ("VLLM_DISABLED_KERNELS", "ConchLinearKernel,MarlinLinearKernel")):+ with pytest.raises(RuntimeError, match=key):+ check_w4a16_environment({key: value})+ monkeypatch.setenv("VLLM_MARLIN_INPUT_DTYPE", "int8")+ owner = SimpleNamespace(device="cuda", repo_dir=tmp_path, config={}, weights="packed",+ _load_lm_weights=mock.Mock())+ with mock.patch("kittenml.kittentts2.vllm.backend.validate_environment"):+ with pytest.raises(RuntimeError, match="W4A8"):+ VLLMBackend(owner, {"quantization": "w4a16"})+ owner._load_lm_weights.assert_not_called()+++def test_w4a16_hf_overrides_payload():+ from kittenml.kittentts2.vllm.backend import quantization_args+ from kittenml.kittentts2.vllm.w4a16 import QUANTIZATION_CONFIG++ assert quantization_args(None) == {}+ args = quantization_args("w4a16")+ assert args["quantization"] == "compressed-tensors"+ payload = args["hf_overrides"]["quantization_config"]+ assert payload == {+ "quant_method": "compressed-tensors", "format": "pack-quantized",+ "config_groups": {"group_0": {+ "targets": ["Linear"],+ "weights": {"num_bits": 4, "type": "int", "symmetric": True, "strategy": "group",+ "group_size": 128, "dynamic": False, "actorder": None},+ "input_activations": None}},+ "ignore": ["lm_head"]}+ payload["total_num_heads"] = 16 # vLLM writes into the dict it receives+ assert "total_num_heads" not in QUANTIZATION_CONFIG+++def test_w4a16_payload_selects_vllm_uint4b8_weight_only_scheme():+ pytest.importorskip("vllm")+ from vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors import (+ CompressedTensorsConfig)+ from vllm.model_executor.layers.quantization.compressed_tensors.schemes.compressed_tensors_wNa16 import (+ WNA16_SUPPORTED_TYPES_MAP)+ from kittenml.kittentts2.vllm.w4a16 import OFFSET, quantization_config++ config = CompressedTensorsConfig.from_config(quantization_config())+ assert config.ignore == ["lm_head"] and config.quant_format == "pack-quantized"+ scheme = config.target_scheme_map["Linear"]+ assert scheme["input_activations"] is None+ assert CompressedTensorsConfig._is_wNa16_group_channel(scheme["weights"], None)+ weights = scheme["weights"]+ assert (weights.num_bits, weights.group_size, weights.symmetric) == (4, 128, True)+ assert WNA16_SUPPORTED_TYPES_MAP[4].bias == OFFSET == 8 # uint4b8: code = sign + 8+++def test_engine_receives_w4a16_arguments_and_main_process_loads_only_conditioning(tmp_path):+ pytest.importorskip("vllm")+ from kittenml.kittentts2.vllm.backend import VLLMBackend++ import os+ seen = []++ def create_engine(**kwargs):+ seen.append(dict(kwargs, disabled_kernels=os.environ.get("VLLM_DISABLED_KERNELS")))++ def run(options):+ owner = SimpleNamespace(device="cuda", repo_dir=tmp_path, config={}, weights="packed",+ token_map=None, _load_lm_weights=mock.Mock(return_value={+ "model.embed_tokens.weight": torch.zeros(2, 4)}))+ with mock.patch("kittenml.kittentts2.vllm.backend.validate_environment"), \+ mock.patch("torch.cuda.get_device_capability", return_value=(8, 6)), \+ mock.patch("transformers.AutoConfig.from_pretrained", return_value=SimpleNamespace(+ model_type="qwen3", hidden_size=4)), \+ mock.patch("transformers.AutoTokenizer.from_pretrained"), \+ mock.patch("kittenml.kittentts2.speaker.load_speaker_head"), \+ mock.patch("vllm.LLM", side_effect=create_engine):+ VLLMBackend(owner, options)+ include = owner._load_lm_weights.call_args.kwargs["include"]+ assert include("model.embed_tokens.weight") and include("spk_proj.0.weight")+ assert not include("model.layers.0.self_attn.q_proj.weight")+ return owner, seen[-1]++ os.environ.pop("VLLM_DISABLED_KERNELS", None)+ owner, kwargs = run(None)+ assert {"quantization", "hf_overrides", "kv_cache_memory_bytes"}.isdisjoint(kwargs)+ assert "quantization" not in kwargs["model_loader_extra_config"]+ assert kwargs["dtype"] is torch.bfloat16 and owner.dtype is torch.bfloat16+ assert kwargs["disabled_kernels"] is None+ owner, kwargs = run({"quantization": "w4a16"})+ # Machete, which vLLM prefers on Hopper, is disabled for the engine only.+ assert kwargs["disabled_kernels"] == "MacheteLinearKernel"+ assert "VLLM_DISABLED_KERNELS" not in os.environ+ owner, kwargs = run({"quantization": "w4a16", "kv_cache_memory_bytes": 1 << 30,+ "dtype": "bfloat16"})+ assert kwargs["quantization"] == "compressed-tensors"+ assert kwargs["hf_overrides"]["quantization_config"]["ignore"] == ["lm_head"]+ assert kwargs["model_loader_extra_config"]["quantization"] == "w4a16"+ assert kwargs["kv_cache_memory_bytes"] == 1 << 30+ assert kwargs["dtype"] is torch.bfloat16 and owner.dtype is torch.bfloat16+++def _linear_with_kernel(kind, act_type=torch.bfloat16):+ """A module carrying a compressed-tensors scheme whose kernel is `kind`."""+ kernel = object.__new__(kind)+ kernel.config = SimpleNamespace(act_type=act_type)+ module = torch.nn.Module()+ module.scheme = SimpleNamespace(kernel=kernel)+ return module+++def test_vllm_loader_streams_w4a16_tensors_without_speaker_head():+ pytest.importorskip("vllm")+ from vllm.model_executor.kernels.linear import MarlinLinearKernel+ from kittenml.kittentts2.vllm.loader import KittenModelLoader++ loader = object.__new__(KittenModelLoader)+ loader.load_config = SimpleNamespace(model_loader_extra_config={+ "repo_dir": "unused", "config": {}, "weights": "packed", "quantization": "w4a16"})++ class TinyModel:+ def load_weights(self, weights):+ self.loaded = dict(weights)+ return set(self.loaded)++ def named_parameters(self):+ return [("model.layers.0.mlp.down_proj.weight_packed", None)]++ def named_modules(self):+ return [("model.layers.0.mlp.down_proj", _linear_with_kernel(MarlinLinearKernel))]++ model = TinyModel()+ tensors = [("model.layers.0.mlp.down_proj.weight_packed", torch.zeros(1, 1, dtype=torch.int32))]+ with mock.patch("kittenml.kittentts2.vllm.w4a16.iter_w4a16",+ return_value=iter(tensors)) as convert:+ loader.load_weights(model, SimpleNamespace(dtype=torch.bfloat16))+ convert.assert_called_once_with("unused", {}, "packed")+ assert set(model.loaded) == {"model.layers.0.mlp.down_proj.weight_packed"}+ loader.load_config.model_loader_extra_config["quantization"] = "int8"+ with pytest.raises(ValueError, match="unknown KittenTTS quantization"):+ loader.load_weights(model, None)+++def test_vllm_loader_allows_only_kv_cache_scales_to_be_missing():+ # compressed-tensors adds KV-cache scale parameters to every attention layer;+ # the checkpoint has none (the KV cache stays in the model dtype). On a GPU,+ # the strict missing-weights check rejected the real model because of them.+ pytest.importorskip("vllm")+ from vllm.model_executor.kernels.linear import MarlinLinearKernel+ from vllm.model_executor.layers.quantization.kv_cache import BaseKVCacheMethod+ from kittenml.kittentts2.vllm.loader import KittenModelLoader++ loader = object.__new__(KittenModelLoader)+ loader.load_config = SimpleNamespace(model_loader_extra_config={+ "repo_dir": "unused", "config": {}, "weights": "packed", "quantization": "w4a16"})+ attn = torch.nn.Module()+ attn.quant_method = object.__new__(BaseKVCacheMethod)+ attn.k_scale = torch.nn.Parameter(torch.tensor(-1.0), requires_grad=False)+ attn.k_zero_point = torch.nn.Parameter(torch.tensor(0), requires_grad=False)+ model = torch.nn.Module()+ model.attn = attn+ model.proj = torch.nn.Linear(2, 2, bias=False)+ model.proj.scheme = _linear_with_kernel(MarlinLinearKernel).scheme+ model.load_weights = lambda weights: {name for name, _ in weights}+ with mock.patch("kittenml.kittentts2.vllm.w4a16.iter_w4a16",+ return_value=iter([("proj.weight", torch.zeros(2, 2))])):+ loader.load_weights(model, SimpleNamespace(dtype=torch.bfloat16))+ with mock.patch("kittenml.kittentts2.vllm.w4a16.iter_w4a16", return_value=iter([])):+ with pytest.raises(ValueError, match=r"missing from the checkpoint: \['proj.weight'\]"):+ loader.load_weights(model, SimpleNamespace(dtype=torch.bfloat16))++[email protected]("variant", ["packed", "full"])+def test_selected_tensor_loading_matches_full_load(tmp_path, variant):+ from kittenml.kittentts2.vllm.backend import _main_process_tensor++ _, config = _w4a16_checkpoint(tmp_path, layers=1)+ everything = load_lm_weights(tmp_path, config, variant)+ selected = load_lm_weights(tmp_path, config, variant, include=_main_process_tensor)+ assert set(selected) == {"model.embed_tokens.weight", "spk_proj.0.weight", "spk_proj.0.bias",+ "spk_proj.1.weight", "spk_proj.1.bias"}+ assert all(torch.equal(selected[k], everything[k]) for k in selected)+++def test_vllm_loader_refuses_w4a16_layers_on_any_kernel_but_marlin():+ # On Hopper vLLM would pick Machete, and with VLLM_DISABLED_KERNELS it can fall+ # back to Triton or others; exact output was verified on Marlin only.+ pytest.importorskip("vllm")+ from vllm.model_executor.kernels.linear import MarlinLinearKernel+ from vllm.model_executor.kernels.linear.mixed_precision.machete import MacheteLinearKernel+ from kittenml.kittentts2.vllm.loader import KittenModelLoader++ loader = object.__new__(KittenModelLoader)+ loader.load_config = SimpleNamespace(model_loader_extra_config={+ "repo_dir": "unused", "config": {}, "weights": "packed", "quantization": "w4a16"})+ model = torch.nn.Module()+ model.a = _linear_with_kernel(MarlinLinearKernel)+ model.b = _linear_with_kernel(MacheteLinearKernel)+ model.load_weights = mock.Mock(return_value=set())+ with mock.patch("kittenml.kittentts2.vllm.w4a16.iter_w4a16") as convert:+ with pytest.raises(RuntimeError, match=r"\['MacheteLinearKernel'\] for 1 linear layers"):+ loader.load_weights(model, SimpleNamespace(dtype=torch.bfloat16))+ with pytest.raises(RuntimeError, match="no quantized linear layers"):+ loader.load_weights(torch.nn.Module(), SimpleNamespace(dtype=torch.bfloat16))+ convert.assert_not_called()+ model.load_weights.assert_not_called()+++def test_vllm_loader_refuses_w4a8_activations():+ # VLLM_MARLIN_INPUT_DTYPE keeps MarlinLinearKernel but sets its activation type+ # to int8/fp8 (W4A8). The loader runs in the engine process and checks the+ # activation type vLLM gave each layer.+ pytest.importorskip("vllm")+ from vllm.model_executor.kernels.linear import MarlinLinearKernel+ from kittenml.kittentts2.vllm.loader import KittenModelLoader++ loader = object.__new__(KittenModelLoader)+ loader.load_config = SimpleNamespace(model_loader_extra_config={+ "repo_dir": "unused", "config": {}, "weights": "packed", "quantization": "w4a16"})+ bf16 = SimpleNamespace(dtype=torch.bfloat16)+ model = torch.nn.Module()+ model.a = _linear_with_kernel(MarlinLinearKernel)+ model.b = _linear_with_kernel(MarlinLinearKernel, act_type=torch.int8)+ model.load_weights = mock.Mock(return_value=set())+ with mock.patch("kittenml.kittentts2.vllm.w4a16.iter_w4a16") as convert:+ with pytest.raises(RuntimeError, match=r"1 linear layers would quantize their "+ r"activations to \['torch.int8'\].*"+ r"VLLM_MARLIN_INPUT_DTYPE"):+ loader.load_weights(model, bf16)+ model.b = _linear_with_kernel(MarlinLinearKernel)+ with pytest.raises(RuntimeError, match="engine dtype is torch.float16"):+ loader.load_weights(model, SimpleNamespace(dtype=torch.float16))+ convert.assert_not_called()+ model.load_weights.assert_not_called()+++def test_engine_sees_the_checked_environment_after_an_earlier_engine(monkeypatch, tmp_path):+ # In single-process mode vLLM freezes its view of the environment once an+ # engine has started (enable_envs_cache). A later W4A16 engine must see the+ # environment KittenTTS checked, not the frozen one.+ pytest.importorskip("vllm")+ import vllm.envs+ from kittenml.kittentts2.vllm.backend import VLLMBackend++ seen = []++ def create_engine(**kwargs):+ seen.append((vllm.envs.VLLM_MARLIN_INPUT_DTYPE, vllm.envs.VLLM_DISABLED_KERNELS))++ owner = SimpleNamespace(device="cuda", repo_dir=tmp_path, config={}, weights="packed",+ token_map=None, _load_lm_weights=mock.Mock(return_value={+ "model.embed_tokens.weight": torch.zeros(2, 4)}))+ monkeypatch.delenv("VLLM_DISABLED_KERNELS", raising=False)+ monkeypatch.setenv("VLLM_MARLIN_INPUT_DTYPE", "int8")+ try:+ vllm.envs.enable_envs_cache() # as after a first engine in this process+ monkeypatch.delenv("VLLM_MARLIN_INPUT_DTYPE")+ assert vllm.envs.VLLM_MARLIN_INPUT_DTYPE == "int8"+ with mock.patch("kittenml.kittentts2.vllm.backend.validate_environment"), \+ mock.patch("torch.cuda.get_device_capability", return_value=(8, 6)), \+ mock.patch("transformers.AutoConfig.from_pretrained", return_value=SimpleNamespace(+ model_type="qwen3", hidden_size=4)), \+ mock.patch("transformers.AutoTokenizer.from_pretrained"), \+ mock.patch("kittenml.kittentts2.speaker.load_speaker_head"), \+ mock.patch("vllm.LLM", side_effect=create_engine):+ VLLMBackend(owner, {"quantization": "w4a16"})+ finally:+ vllm.envs.disable_envs_cache()+ assert seen == [(None, ["MacheteLinearKernel"])]+++def test_w4a16_exactness_checks_compare_bits_not_values(tmp_path):+ # torch.equal treats -0.0 and +0.0 as equal; the W4A16 checks must not.+ from kittenml.kittentts2.vllm.w4a16 import bit_equal, iter_w4a16++ zero = torch.zeros(3, dtype=torch.bfloat16)+ assert torch.equal(-zero, zero) and not bit_equal(-zero, zero)+ assert bit_equal(zero, zero.clone()) and not bit_equal(zero, zero.float())+ weight = torch.zeros(2, 128, dtype=torch.bfloat16)+ weight[0, :2] = torch.tensor([0.5, -0.5])+ weight[1, 3] = -0.0 # a ternary rebuild gives +0.0 here+ (tmp_path / "lm").mkdir()+ save_file({"model.layers.0.self_attn.o_proj.weight": weight},+ tmp_path / "lm" / "model.safetensors")+ with pytest.raises(ValueError, match="not exactly ternary"):+ list(iter_w4a16(tmp_path, {}, "full"))+ weight[1, 3] = 0.0+ save_file({"model.layers.0.self_attn.o_proj.weight": weight},+ tmp_path / "lm" / "model.safetensors")+ assert len(list(iter_w4a16(tmp_path, {}, "full"))) == 3