Skip to content

Open-source State-of-the-art TTS model which runs on a CPU 😻

Python

Add exact 4-bit (W4A16) weights to the vLLM backend

haaziq-sl47e02b9a002c
diff --git a/README.md b/README.md
index 0667e9a..0b9c897 100644
--- a/README.md
+++ b/README.md
@@ -213,7 +213,8 @@ print(model.last_report) # Generation timings and token IDs.
## Running with vLLM
-For GPU inference with vLLM on Linux and an NVIDIA CUDA GPU, install the optional backend:
+For GPU inference with vLLM on Linux and an NVIDIA CUDA GPU with compute capability
+8.0 or newer (Ampere or later, for bfloat16), install the optional backend:
```bash
pip install "kittenml[vllm]"
@@ -225,7 +226,8 @@ Select `backend="vllm"`:
from kittenml import KittenTTS
def main():
- m = KittenTTS("KittenML/kitten-tts-2", backend="vllm")
+ m = KittenTTS("KittenML/kitten-tts-2", backend="vllm",
+ vllm_options={"quantization": "w4a16"})
m.generate_to_file("Hello there.", "output.wav", voice="Bruno")
print("Saved output.wav")
@@ -234,7 +236,10 @@ if __name__ == "__main__":
```
Keep the main guard when running a script: vLLM starts a worker process. The same
-voices, decoders, weight choices and generation arguments apply. See the
+voices, decoders, weight choices and generation arguments apply. The example
+uses 4-bit language-model weights (`"quantization": "w4a16"`): the same weights, exactly, in about
+2 GB less GPU memory, and faster for single requests. Leave out `vllm_options` to run in bf16; see
+[4-bit weights](docs/vllm.md#4-bit-weights). See the
[vLLM guide](docs/vllm.md) for memory settings and benchmarks.
diff --git a/docs/vllm.md b/docs/vllm.md
index b0d32c6..334f594 100644
--- a/docs/vllm.md
+++ b/docs/vllm.md
@@ -1,72 +1,103 @@
# Running with vLLM
-The optional `vllm` backend runs KittenTTS 2's language model on an NVIDIA CUDA
-GPU. Audio decoding, voice cloning and text preparation use the existing
-KittenTTS implementations. The default backend remains PyTorch and already
-supports CUDA.
-
-Install the optional backend and select `backend="vllm"` as shown in the
-[README](../README.md#running-with-vllm). This backend requires Linux and uses
-vLLM 0.31.0; see its [GPU requirements](https://docs.vllm.ai/en/v0.31.0/getting_started/installation/gpu/)
-for compatible NVIDIA drivers.
-It loads the selected published weights directly; no manual export
-or additional language-model checkpoint is required. Packed weights are expanded in
-memory, so their smaller download size does not reduce GPU memory use.
-
-The engine defaults to a 4,096-token context, up to eight active sequences, and
-half the GPU memory, leaving space for audio decoding. In the README example,
-pass `vllm_options` when constructing the model to change these settings:
-
-```python
-m = KittenTTS("KittenML/kitten-tts-2", backend="vllm",
- vllm_options={"gpu_memory_utilization": 0.5,
- "max_model_len": 4096})
+The optional `vllm` backend runs KittenTTS 2's language model with vLLM 0.31.0 on an NVIDIA GPU;
+audio decoding, voice cloning and text preparation stay the same. It needs Linux and a GPU with
+compute capability 8.0 or newer (Ampere or later); older GPUs are refused before any download.
+
+Install and select it as shown in the [README](../README.md#running-with-vllm). That install gets
+CUDA 13.0 builds of vLLM and PyTorch, which need a driver for CUDA 13.0
+([GPU requirements](https://docs.vllm.ai/en/v0.31.0/getting_started/installation/gpu/)). If
+`nvidia-smi` reports a lower CUDA version, install the CUDA 12.9 builds with
+[uv](https://docs.astral.sh/uv/) instead (checked with driver 570.195, which reports CUDA 12.8):
+
+```bash
+curl -LO https://github.com/vllm-project/vllm/releases/download/v0.31.0/vllm-0.31.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl
+uv pip install --torch-backend cu129 "kittenml[vllm]" ./vllm-0.31.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl
```
-`gpu_memory_utilization` is a fraction of the GPU's total VRAM. It sets vLLM's
-budget; the audio decoder and other GPU processes need additional memory.
+## Options
+
+| Option | Default | What it is for |
+|---|---|---|
+| `gpu_memory_utilization` | `0.5` | Share of the GPU's memory vLLM may use for the language model and its cache. Audio decoding needs memory outside this share. |
+| `max_model_len` | `4096` | Longest request in tokens: the voice's reference prompt and the text, plus `max_new_tokens` (1000 by default). A request that does not fit raises an error. |
+| `max_num_seqs` | `8` | Most requests generated at the same time. |
+| `kv_cache_memory_bytes` | not set | A fixed cache size in bytes, used instead of sizing the cache from `gpu_memory_utilization`. |
+| `enforce_eager` | `False` | Turns off CUDA graphs. Keep it off: with it on, 4-bit can be slower than the default bf16. |
+| `quantization` | not set | `"w4a16"` for 4-bit weights; see [4-bit weights](#4-bit-weights). |
+| `dtype` | `"bfloat16"` | Precision of the language model. Only `"bfloat16"` is accepted; `"float16"` is refused. |
+
+Pass these as `vllm_options` when constructing the model, for example
+`vllm_options={"max_model_len": 2048}`; unknown option names are refused before any download. Select
+the GPU with `CUDA_VISIBLE_DEVICES`; set `VLLM_LOGGING_LEVEL=INFO` before starting Python to see
+routine engine logs. Long text uses the default backend's sentence chunking, and `generate_stream`
+yields each audio chunk once it is complete. The same seed can give different audio across backends
+and with or without 4-bit weights. Start-up is slow (see [Benchmarks](#benchmarks)), so reuse the
+loaded model.
-If VRAM is limited, reduce `max_model_len` to lower cache requirements. If startup
-reports no available memory for cache blocks, increase `gpu_memory_utilization`
-to give vLLM a larger share of VRAM, leaving space for audio decoding and other
-GPU processes.
+## Low GPU memory
-Routine engine logs are quiet by default; warnings and errors remain visible.
-Set `VLLM_LOGGING_LEVEL=INFO` before starting Python to see startup diagnostics.
+- Reduce `max_model_len`, keeping room for the prompt plus `max_new_tokens`: a shorter context
+ needs a smaller cache.
+- If startup reports no memory available for the cache, raise `gpu_memory_utilization`, leaving room
+ for audio decoding, or set `kv_cache_memory_bytes`. The cache needs 112 KiB per token; a cache of
+ exactly `max_model_len` tokens is rejected, so leave some margin.
+- Use [4-bit weights](#4-bit-weights); the smaller `weights="packed"` download alone does not
+ lower GPU memory.
+- Cloning without `reference_text` also loads Whisper large-v3 on the GPU to transcribe the clip;
+ pass `reference_text` to skip it.
-The context must fit the reference prompt and the requested generation budget.
-Long text still uses KittenTTS's sentence chunking. `generate_stream` yields
-completed audio chunks, just like the default backend.
+## 4-bit weights
-`vllm_options` also accepts `max_num_seqs` and `enforce_eager`. Select a GPU with
-`CUDA_VISIBLE_DEVICES` before starting Python. Sampling implementations differ
-between the backends, so a matching seed does not guarantee identical audio.
+Add `"quantization": "w4a16"` to `vllm_options` to use about 2 GB less GPU memory for the language
+model's weights (1.29 instead of 3.24 GiB). The model's weights stay exactly the same, and measured
+audio quality was the same as without it. With default settings vLLM still takes half the GPU and
+gives the saved memory to its cache; to lower total use, also set `kv_cache_memory_bytes` or a lower
+`gpu_memory_utilization`. It works with `weights="packed"` (the default) or `"full"`, and also
+with `weights="emb4"`. That one is lossy: its embedding is stored in 4 bits (at export, perplexity
+13.60 against 13.50), and it only makes the download smaller, not GPU memory, because the
+embedding is expanded to bf16 on the GPU either way. KittenTTS checks that vLLM runs the 4-bit layers
+with the kernel they were verified with, and refuses settings that would change it.
+
+The biggest gain is with one request at a time: the language model ran about 1.4x faster on an RTX
+PRO 6000 (96 GB) and 1.8x faster on an RTX A2000 (6 GB). With many concurrent requests the gain is
+small (1.02x with 50 at once on the RTX PRO 6000). On the RTX A2000, bf16 does not start with
+default `vllm_options` but 4-bit does; with `kv_cache_memory_bytes` 512 MiB and `max_num_seqs` 1 for
+both, five sentences one at a time:
+
+| RTX A2000 (6 GB) | bf16 (default) | 4-bit |
+|---|---:|---:|
+| Peak GPU memory, whole card | 6,116 MiB | 4,096 MiB |
+| Median time per sentence | 2.65 s | 1.73 s |
+| Language-model speed | 54.5 tokens/s | 97.2 tokens/s |
## Benchmarks
-Measured on an RTX 3060 (12 GB), Linux, Python 3.12 and NVIDIA driver 580.95.05,
-using `weights="packed"`, the default decoder and the stable preset. These are
-median end-to-end synthesis times from five runs after two warmups per voice.
-Short, medium and long inputs contain 61, 226 and 680 characters; the long input
-is split into two chunks. Model initialization and file writes are excluded.
+One RTX 3060 (12 GB), driver 570.195.03 (CUDA 12.8), Ubuntu 24.04, Python 3.12, nothing else on the
+GPU. PyTorch GPU is the default backend (PyTorch 2.14.1 CUDA 12.6 build, Transformers 5.20.2); the
+vLLM columns use vLLM 0.31.0 (CUDA 12.9 build, PyTorch 2.13.0, Transformers 5.17.0) with default
+`vllm_options`: "vLLM bf16" is without quantization, "vLLM 4-bit" adds `"quantization": "w4a16"`
+([4-bit weights](#4-bit-weights)). Each time is the median of five `generate()` calls after two
+warmups (`weights="packed"`, default decoder, stable preset, model loading excluded) on 61, 226 and
+680 characters of text (the last in two chunks).
-| Input / voice | PyTorch GPU | vLLM | Speedup |
+| Input / voice | PyTorch GPU | vLLM bf16 | vLLM 4-bit | Speedup bf16 vs PyTorch | Speedup 4-bit vs PyTorch |
+|---|---:|---:|---:|---:|---:|
+| Short / Bruno | 4.44 s | 1.72 s | 1.18 s | 2.59x | 3.77x |
+| Short / Bella | 6.14 s | 2.67 s | 1.62 s | 2.30x | 3.78x |
+| Medium / Bruno | 14.74 s | 5.12 s | 3.43 s | 2.88x | 4.30x |
+| Medium / Bella | 20.63 s | 7.01 s | 4.63 s | 2.94x | 4.46x |
+| Long / Bruno | 45.20 s | 16.10 s | 10.38 s | 2.81x | 4.35x |
+| Long / Bella | 53.15 s | 18.96 s | 12.24 s | 2.80x | 4.34x |
+
+| RTX 3060 (12 GB) | PyTorch GPU | vLLM bf16 | vLLM 4-bit |
|---|---:|---:|---:|
-| Short / Bruno | 6.01 s | 2.05 s | 2.93x |
-| Short / Bella | 6.98 s | 2.54 s | 2.75x |
-| Medium / Bruno | 22.93 s | 5.49 s | 4.18x |
-| Medium / Bella | 27.95 s | 8.26 s | 3.39x |
-| Long / Bruno | 56.77 s | 17.56 s | 3.23x |
-| Long / Bella | 79.24 s | 24.04 s | 3.30x |
-
-The PyTorch column uses unchanged upstream code in a fresh normal installation
-(PyTorch 2.14.1, Transformers 5.18.0). With versions matched to the vLLM install
-(PyTorch 2.13.0, Transformers 5.17.0, vLLM 0.31.0), the measured speedup was
-3.05-4.49x. Generated audio lengths differ between samplers; vLLM's synthesis
-time was 0.44-0.48 seconds per second of output audio in these tests.
-
-vLLM used about 8.5 GiB of GPU memory versus 5.8 GiB for the normal installation.
-Cloning without `reference_text` also loads Whisper; those tests used up to
-11.5 GiB. Supplying the transcript skips that model.
-Initialization with cached downloads took 51 seconds versus 10 seconds; the
-first vLLM startup took about 162 seconds. Reuse the loaded model across calls.
+| Synthesis time per second of audio | 1.17-1.19 s | 0.41-0.45 s | 0.26-0.30 s |
+| Peak GPU memory, whole card | 6,901 MiB | 9,026 MiB | 9,890 MiB |
+| Start-up with files already downloaded (very first start) | 9.8 s (11.2 s) | 44.0 s (145.4 s) | 50.3 s (140.7 s) |
+
+Per second of audio the speedups are 2.64-2.85x (bf16) and 3.94-4.51x (4-bit). Both vLLM columns
+give vLLM the same share of the card, so 4-bit gets a larger cache (37,120 tokens instead of
+18,512); its higher whole-card peak was in the main process, which decodes the audio, not in
+vLLM's own process. Whole-card peaks include about 374 MiB the GPU reports as used when idle. A
+full repeat of every column was within -0.9% to +3.4%.
diff --git a/kittenml/get_model.py b/kittenml/get_model.py
index b76afa1..284de9b 100644
--- a/kittenml/get_model.py
+++ b/kittenml/get_model.py
@@ -71,8 +71,11 @@ class KittenTTS(KittenTTSOnnx):
if model_type == KITTEN2_TYPE:
from .kittentts2 import KittenTTS2
if backend == "vllm":
- from .kittentts2.vllm.backend import validate_environment
+ from .kittentts2.vllm.backend import check_options, validate_environment
validate_environment(device)
+ # Before the download: an unsupported GPU or a misspelled option
+ # should not cost gigabytes.
+ check_options(vllm_options, weights or config.get("default_weights", "packed"))
elif vllm_options is not None:
raise ValueError("vllm_options requires backend='vllm'")
if local_dir is None:
diff --git a/kittenml/kittentts2/model.py b/kittenml/kittentts2/model.py
index 8d356ee..4a80b3e 100644
--- a/kittenml/kittentts2/model.py
+++ b/kittenml/kittentts2/model.py
@@ -248,16 +248,18 @@ class KittenTTS2:
from .weights import available_weights
return available_weights(self.config)
- def _load_lm_weights(self, lm_dir):
+ def _load_lm_weights(self, lm_dir, include=None):
"""The language model's weights, in whichever packing was selected.
"packed" is a 1.58-bit packing that is exactly equivalent to
model.safetensors at about a third of the size, and is the default.
"emb4" is smaller again but quantises the embedding, which is lossy β€”
- see tl2_emb4.py. "full" reads the plain bf16 files.
+ see tl2_emb4.py. "full" reads the plain bf16 files. `include`, a
+ predicate on tensor names, limits loading to the tensors it accepts.
"""
from .weights import load_lm_weights
- return load_lm_weights(self.repo_dir, self.config, self.weights, lm_dir=lm_dir)
+ return load_lm_weights(self.repo_dir, self.config, self.weights, lm_dir=lm_dir,
+ include=include)
def _load_voices(self):
voices_path = self.repo_dir / self.config.get("voices", "voices/voices.json")
diff --git a/kittenml/kittentts2/vllm/backend.py b/kittenml/kittentts2/vllm/backend.py
index 25deebb..024c1ea 100644
--- a/kittenml/kittentts2/vllm/backend.py
+++ b/kittenml/kittentts2/vllm/backend.py
@@ -10,16 +10,23 @@ import torch
@contextlib.contextmanager
-def _engine_environment():
+def _engine_environment(forced=None):
# KittenTTS already applies its warpers. Use vLLM's native sampler by default
# so a pip installation needs no nvcc for FlashInfer's sampling JIT. The
# worker inherits these settings, including quiet startup defaults. Restore
- # the caller's environment afterwards and respect explicit overrides.
+ # the caller's os.environ afterwards and respect explicit overrides.
+ # `forced` values replace the caller's while the engine is created. vLLM reads
+ # them when it builds the engine; in single-process mode
+ # (VLLM_ENABLE_V1_MULTIPROCESSING=0) it then keeps its own frozen copy of the
+ # environment for the rest of the process, which restoring os.environ does
+ # not change (VLLMBackend drops that copy before creating each engine).
defaults = {"VLLM_USE_FLASHINFER_SAMPLER": "0",
"VLLM_LOGGING_LEVEL": "WARNING", "TQDM_DISABLE": "1"}
- previous = {key: os.environ.get(key) for key in defaults}
+ forced = dict(forced or {})
+ previous = {key: os.environ.get(key) for key in {**defaults, **forced}}
for key, value in defaults.items():
os.environ.setdefault(key, value)
+ os.environ.update(forced)
try:
yield
finally:
@@ -40,6 +47,102 @@ def validate_environment(device=None):
raise RuntimeError("the vLLM backend requires an NVIDIA CUDA GPU")
if selected.index not in (None, 0):
raise ValueError("select the GPU with CUDA_VISIBLE_DEVICES and use device='cuda'")
+ major, minor = torch.cuda.get_device_capability()
+ if major < 8:
+ raise RuntimeError(f"this GPU (SM{major}{minor}) has no bfloat16 support, which "
+ f"KittenTTS's language model needs: {_NO_FLOAT16}")
+
+
+# Engine settings KittenTTS chooses for vLLM, and their defaults.
+_ENGINE_DEFAULTS = dict(gpu_memory_utilization=0.5, max_model_len=4096,
+ max_num_seqs=8, enforce_eager=False)
+# Options KittenTTS interprets itself. None keeps the default behaviour.
+_KITTEN_OPTIONS = ("quantization", "dtype", "kv_cache_memory_bytes")
+_QUANTIZATIONS = (None, "w4a16")
+_DTYPES = {None: None, "bfloat16": torch.bfloat16}
+# W4A16 runs on vLLM's Marlin kernel, the one whose exact output was verified.
+# vLLM tries Machete first on Hopper (SM90) GPUs, so it is disabled for the
+# engine; the loader then checks that every quantized layer uses Marlin.
+_W4A16_KERNEL = "MarlinLinearKernel"
+_W4A16_DISABLED_KERNELS = ("MacheteLinearKernel",)
+# Measured with the current checkpoint, with and without W4A16: in float16 the
+# language model's residual stream passes float16's maximum (65,504) at layer 10
+# and overflows, after which it emits no audio tokens.
+_NO_FLOAT16 = ("this model's residual stream exceeds the float16 range (measured overflow "
+ "from layer 10); GPUs without bfloat16 (compute capability < 8.0) are not "
+ "supported by the vLLM backend")
+
+
+def resolve_options(options):
+ """Split `vllm_options` into engine keyword arguments and KittenTTS choices."""
+ options = dict(options or {})
+ unknown = set(options) - _ENGINE_DEFAULTS.keys() - set(_KITTEN_OPTIONS)
+ if unknown:
+ raise ValueError(f"unknown vllm_options: {sorted(unknown)}")
+ kitten = {key: options.pop(key, None) for key in _KITTEN_OPTIONS}
+ if kitten["quantization"] not in _QUANTIZATIONS:
+ raise ValueError(f"vllm_options['quantization'] must be None or 'w4a16', "
+ f"not {kitten['quantization']!r}")
+ if kitten["dtype"] == "float16":
+ raise ValueError(f"vllm_options['dtype']='float16' is not supported: {_NO_FLOAT16}")
+ if kitten["dtype"] not in _DTYPES:
+ raise ValueError(f"vllm_options['dtype'] must be None or 'bfloat16', "
+ f"not {kitten['dtype']!r}")
+ memory = kitten["kv_cache_memory_bytes"]
+ if memory is not None and (isinstance(memory, bool) or not isinstance(memory, int)
+ or memory <= 0):
+ raise ValueError("vllm_options['kv_cache_memory_bytes'] must be a positive integer")
+ engine = dict(_ENGINE_DEFAULTS)
+ engine.update(options)
+ if memory is not None:
+ engine["kv_cache_memory_bytes"] = memory
+ return engine, kitten
+
+
+def check_options(options, weights):
+ """Validate `vllm_options` before anything is downloaded or loaded."""
+ defaults, kitten = resolve_options(options)
+ if kitten["quantization"] == "w4a16":
+ check_w4a16_environment()
+ return defaults, kitten
+
+
+def check_w4a16_environment(environ=None):
+ """Refuse settings that would change what W4A16 computes or which kernel runs it."""
+ environ = os.environ if environ is None else environ
+ refuse = "vllm_options['quantization']='w4a16' refuses to start while "
+ if "VLLM_MARLIN_INPUT_DTYPE" in environ:
+ raise RuntimeError(refuse + "VLLM_MARLIN_INPUT_DTYPE is set: it quantizes activations "
+ "to int8/fp8, turning W4A16 into W4A8. Unset it to run exact W4A16")
+ if environ.get("VLLM_MARLIN_USE_ATOMIC_ADD") == "1": # vLLM reads only "1" as on
+ raise RuntimeError(refuse + "VLLM_MARLIN_USE_ATOMIC_ADD=1: it would let Marlin reduce "
+ "with unordered atomic adds. Unset it to run exact W4A16")
+ disabled = environ.get("VLLM_DISABLED_KERNELS", "").split(",")
+ if _W4A16_KERNEL in disabled:
+ raise RuntimeError(refuse + f"VLLM_DISABLED_KERNELS names {_W4A16_KERNEL}, the only "
+ "kernel exact W4A16 is verified with. Remove it from the list")
+
+
+def w4a16_engine_environment(environ=None):
+ """VLLM_DISABLED_KERNELS for the engine: the caller's list plus Machete."""
+ environ = os.environ if environ is None else environ
+ disabled = [k for k in environ.get("VLLM_DISABLED_KERNELS", "").split(",") if k]
+ disabled += [k for k in _W4A16_DISABLED_KERNELS if k not in disabled]
+ return {"VLLM_DISABLED_KERNELS": ",".join(disabled)}
+
+
+def _main_process_tensor(name):
+ """Only these LM tensors are used outside vLLM: prompt embeddings and spk_proj."""
+ return name == "model.embed_tokens.weight" or name.startswith("spk_proj.")
+
+
+def quantization_args(quantization):
+ """Engine arguments that select exact W4A16 (compressed-tensors, Marlin)."""
+ if quantization is None:
+ return {}
+ from .w4a16 import quantization_config
+ return {"quantization": "compressed-tensors",
+ "hf_overrides": {"quantization_config": quantization_config()}}
class VLLMBackend:
@@ -48,40 +151,52 @@ class VLLMBackend:
from transformers import AutoConfig, AutoTokenizer
from ..speaker import load_speaker_head
- options = dict(options or {})
- defaults = dict(gpu_memory_utilization=0.5, max_model_len=4096,
- max_num_seqs=8, enforce_eager=False)
- unknown = set(options) - defaults.keys()
- if unknown:
- raise ValueError(f"unknown vllm_options: {sorted(unknown)}")
- defaults.update(options)
+ defaults, kitten = check_options(options, owner.weights)
+ quantization = kitten["quantization"]
+ forced_environment = w4a16_engine_environment() if quantization == "w4a16" else None
+ default_dtype = (torch.bfloat16 if owner.config.get("dtype", "bf16") == "bf16"
+ else torch.float32)
+ dtype = _DTYPES[kitten["dtype"]] or default_dtype
+ if quantization == "w4a16" and dtype is not torch.bfloat16:
+ raise ValueError(f"W4A16 needs bfloat16 activations, not {dtype}")
self.max_model_len = defaults["max_model_len"]
self.token_map = owner.token_map
lm_dir = owner.repo_dir / owner.config.get("lm_dir", "lm")
config = AutoConfig.from_pretrained(str(lm_dir))
if config.model_type != "qwen3":
raise ValueError("the vLLM backend currently supports Qwen3 KittenTTS 2 checkpoints")
- owner.dtype = torch.bfloat16 if owner.config.get("dtype", "bf16") == "bf16" else torch.float32
+ owner.dtype = dtype
owner.tokenizer = AutoTokenizer.from_pretrained(str(lm_dir))
- state = owner._load_lm_weights(lm_dir)
+ # vLLM's own loader reads the full LM; this process only needs the
+ # embedding and the speaker head, so do not expand the rest here.
+ state = owner._load_lm_weights(lm_dir, include=_main_process_tensor)
self.embedding = state["model.embed_tokens.weight"].to(owner.dtype)
self.spk_proj = load_speaker_head(config.hidden_size, str(lm_dir),
owner.device, owner.dtype,
spk_dim=owner.config.get("speaker_embedding_dim", 512),
state=state)
del state
- with _engine_environment():
+ loader_config = {"repo_dir": str(owner.repo_dir), "config": owner.config,
+ "weights": owner.weights}
+ if quantization is not None:
+ loader_config["quantization"] = quantization
+ with _engine_environment(forced_environment):
+ import vllm.envs
from vllm import LLM
from vllm.config import LoggingConfig
+ # After an engine has started in this process (single-process mode),
+ # vLLM answers from a frozen copy of the environment; drop it so this
+ # engine sees the settings checked above.
+ vllm.envs.disable_envs_cache()
self.engine = LLM(
model=str(lm_dir), tokenizer=str(lm_dir), dtype=owner.dtype,
load_format="kittenml",
- model_loader_extra_config={"repo_dir": str(owner.repo_dir),
- "config": owner.config, "weights": owner.weights},
+ model_loader_extra_config=loader_config,
enable_prompt_embeds=True, enable_prefix_caching=False,
logits_processors=["kittenml.kittentts2.vllm.logits:KittenLogitsProcessor"],
- logging_config=LoggingConfig(), disable_log_stats=True, **defaults)
+ logging_config=LoggingConfig(), disable_log_stats=True,
+ **quantization_args(quantization), **defaults)
def prepare_request(self, prompt, speaker_embedding, settings, max_new_tokens, advanced):
from vllm import SamplingParams
diff --git a/kittenml/kittentts2/vllm/loader.py b/kittenml/kittentts2/vllm/loader.py
index 96b01f7..3d0fd06 100644
--- a/kittenml/kittentts2/vllm/loader.py
+++ b/kittenml/kittentts2/vllm/loader.py
@@ -12,14 +12,90 @@ class KittenModelLoader(BaseModelLoader):
def load_weights(self, model, model_config):
options = self.load_config.model_loader_extra_config
- state = load_lm_weights(options["repo_dir"], options["config"], options["weights"])
- loaded = model.load_weights((name, tensor) for name, tensor in state.items()
- if not name.startswith("spk_proj."))
- missing = set(dict(model.named_parameters())) - loaded
+ quantization = options.get("quantization")
+ if quantization == "w4a16":
+ # vLLM chose each layer's kernel when it built the model; check before loading.
+ _check_w4a16_kernels(model, model_config)
+ # Exact 4-bit weights straight from the trits, one tensor at a time.
+ from .w4a16 import iter_w4a16
+ weights = iter_w4a16(options["repo_dir"], options["config"], options["weights"])
+ elif quantization is None:
+ state = load_lm_weights(options["repo_dir"], options["config"], options["weights"])
+ weights = ((name, tensor) for name, tensor in state.items()
+ if not name.startswith("spk_proj."))
+ else:
+ raise ValueError(f"unknown KittenTTS quantization {quantization!r}")
+ loaded = model.load_weights(weights)
+ missing = set(dict(model.named_parameters())) - loaded - _kv_cache_scale_names(model)
if missing:
raise ValueError(f"weights missing from the checkpoint: {sorted(missing)[:5]}")
+def _quantized_layers(model):
+ """(name, module, kernel) for every linear layer vLLM built with a mixed-precision kernel."""
+ for name, module in model.named_modules():
+ kernel = getattr(getattr(module, "scheme", None), "kernel", None)
+ if kernel is not None:
+ yield name, module, kernel
+
+
+def _check_w4a16_kernels(model, model_config):
+ """Every quantized linear layer must run exact W4A16 on Marlin.
+
+ vLLM picks the mixed-precision kernel per GPU (Machete on Hopper, Marlin
+ elsewhere, others when a kernel is disabled). Exact W4A16 was verified with
+ Marlin only, so refuse to load if vLLM chose anything else. This runs in the
+ engine process: VLLM_MARLIN_INPUT_DTYPE keeps the Marlin kernel but quantizes
+ its activations to 8 bits (W4A8), which shows in the kernel's activation type
+ (in vLLM 0.31.0 that is the only way the setting acts).
+ """
+ import torch
+ from vllm.model_executor.kernels.linear import MarlinLinearKernel
+
+ refuse = "W4A16: refusing to load: "
+ if model_config.dtype != torch.bfloat16:
+ raise RuntimeError(refuse + f"the engine dtype is {model_config.dtype}, not bfloat16")
+ chosen, activations = {}, {}
+ for name, _, kernel in _quantized_layers(model):
+ chosen.setdefault(type(kernel).__name__, []).append(name)
+ act_type = getattr(getattr(kernel, "config", None), "act_type", None)
+ activations.setdefault(str(act_type), []).append(name)
+ if not chosen:
+ raise RuntimeError("W4A16: vLLM created no quantized linear layers")
+ other = {kind: names for kind, names in chosen.items()
+ if kind != MarlinLinearKernel.__name__}
+ if other:
+ layers = sum(map(len, other.values()))
+ raise RuntimeError(f"W4A16: vLLM chose {sorted(other)} for {layers} linear layers, but "
+ "exact W4A16 is verified only with MarlinLinearKernel (check "
+ "VLLM_DISABLED_KERNELS; with VLLM_ENABLE_V1_MULTIPROCESSING=0 vLLM "
+ "may still use the value from the first engine in this process)")
+ wrong = {act: names for act, names in activations.items() if act != str(model_config.dtype)}
+ if wrong:
+ layers = sum(map(len, wrong.values()))
+ raise RuntimeError(refuse + f"{layers} linear layers would quantize their activations to "
+ f"{sorted(wrong)} instead of computing in {model_config.dtype} (check "
+ "VLLM_MARLIN_INPUT_DTYPE)")
+
+
+def _kv_cache_scale_names(model):
+ """Attention KV-cache scales a quantization method adds, which the checkpoint omits.
+
+ With quantization="compressed-tensors", every attention layer gets q/k/v scale
+ and zero-point parameters for a quantized KV cache. KittenTTS keeps the KV cache
+ in the model dtype, so they keep vLLM's defaults (as with vLLM's own loader);
+ every other parameter must still come from the checkpoint.
+ """
+ from vllm.model_executor.layers.quantization.kv_cache import BaseKVCacheMethod
+
+ names = set()
+ for prefix, module in model.named_modules():
+ if isinstance(getattr(module, "quant_method", None), BaseKVCacheMethod):
+ names.update(f"{prefix}.{name}" if prefix else name
+ for name, _ in module.named_parameters(recurse=False))
+ return names
+
+
def register():
from vllm.model_executor.model_loader import register_model_loader
register_model_loader("kittenml")(KittenModelLoader)
diff --git a/kittenml/kittentts2/vllm/w4a16.py b/kittenml/kittentts2/vllm/w4a16.py
new file mode 100644
index 0000000..117e9d9
--- /dev/null
+++ b/kittenml/kittentts2/vllm/w4a16.py
@@ -0,0 +1,227 @@
+"""Exact 4-bit (W4A16) weights for the vLLM backend.
+
+The language model's 196 linear projections are ternary: every 128-wide input
+group of every output row holds only {-s, 0, +s}, with one bf16 scale per group
+(see ternary.py). That is exactly what a symmetric 4-bit, group-128 weight can
+represent: compressed-tensors' ``uint4b8`` type stores ``q = sign + 8``, so the
+codes are 7, 8 and 9, and dequantising gives ``(q - 8) * s``, the original bf16
+weight bit for bit. The group scales are copied through unchanged.
+
+vLLM then runs these layers with its Marlin mixed-precision kernel: 4-bit
+weights, bf16 activations (the backend and loader keep vLLM from choosing
+another kernel or 8-bit activations). Everything that is not ternary stays at
+its stored precision: the token embedding and the tied output head, all norms,
+and the KV cache. ``spk_proj`` runs outside vLLM and is dropped here.
+
+The lossy ``emb4`` packing is refused: W4A16 is offered as an exact
+representation of the published weights and is never combined with it.
+
+vLLM fuses q/k/v into ``qkv_proj`` and gate/up into ``gate_up_proj``
+(``Qwen3ForCausalLM.packed_modules_mapping``), so the 196 source projections
+(7 per layer) become 4 quantized modules per layer: 112 for the 28-layer model.
+
+This module does not import vLLM, so it can be tested on any machine.
+"""
+
+import copy
+import json
+import re
+
+import torch
+
+from .. import ternary
+from ..weights import plain_shards, weights_path
+
+NUM_BITS = 4
+GROUP_SIZE = ternary.GROUP_SIZE # 128, as in the checkpoint
+OFFSET = 1 << (NUM_BITS - 1) # 8: uint4b8 stores sign + 8
+PACK_FACTOR = 32 // NUM_BITS # 8 codes per int32
+
+# The compressed-tensors config vLLM reads from hf_config.quantization_config.
+# Targets match vLLM's linear classes by name (QKVParallelLinear,
+# MergedColumnParallelLinear, RowParallelLinear); the vocabulary embedding is
+# not a Linear and stays unquantized, and the tied head is ignored explicitly.
+QUANTIZATION_CONFIG = {
+ "quant_method": "compressed-tensors",
+ "format": "pack-quantized",
+ "config_groups": {
+ "group_0": {
+ "targets": ["Linear"],
+ "weights": {
+ "num_bits": NUM_BITS,
+ "type": "int",
+ "symmetric": True,
+ "strategy": "group",
+ "group_size": GROUP_SIZE,
+ "dynamic": False,
+ "actorder": None,
+ },
+ "input_activations": None,
+ },
+ },
+ "ignore": ["lm_head"],
+}
+
+# The checkpoint's ternary projections, by name. Anything else passes through.
+_PROJECTION = re.compile(
+ r"^model\.layers\.(\d+)\.(self_attn\.(?:q|k|v|o)_proj|mlp\.(?:gate|up|down)_proj)\.weight$")
+
+def quantization_config():
+ """A fresh copy: vLLM adds keys (head counts) to the dict it is given."""
+ return copy.deepcopy(QUANTIZATION_CONFIG)
+
+
+def is_projection(name):
+ """True for the q/k/v/o/gate/up/down weights that become 4-bit."""
+ return _PROJECTION.match(name) is not None
+
+
+def pack_int4(signs):
+ """int8 values in [-8, 7] -> int32 words, compressed-tensors' bit order.
+
+ Matches ``compressed_tensors.compressors.pack_quantized.helpers.pack_to_int32``
+ with ``num_bits=4, packed_dim=1``: each value is shifted to unsigned
+ (+8), and column ``8*j + i`` of a row occupies bits ``4*i`` to ``4*i + 3`` of
+ word ``j``, lowest column in the lowest bits.
+ """
+ if signs.dtype != torch.int8 or signs.dim() != 2:
+ raise ValueError("expected a 2-D int8 tensor")
+ rows, cols = signs.shape
+ if cols % PACK_FACTOR:
+ raise ValueError(f"input width {cols} is not a multiple of {PACK_FACTOR}")
+ codes = (signs.to(torch.int64) + OFFSET).reshape(rows, cols // PACK_FACTOR, PACK_FACTOR)
+ if codes.min() < 0 or codes.max() >= 1 << NUM_BITS:
+ raise ValueError("values do not fit in 4 bits")
+ shifts = torch.arange(PACK_FACTOR, dtype=torch.int64) * NUM_BITS
+ words = (codes << shifts).sum(dim=2)
+ # Reinterpret the unsigned 32-bit words as int32 without relying on overflow.
+ words = torch.where(words >= 1 << 31, words - (1 << 32), words)
+ return words.to(torch.int32).contiguous()
+
+
+def bit_equal(a, b):
+ """Same dtype, shape and bit pattern. Unlike torch.equal, -0.0 differs from +0.0."""
+ if a.dtype != b.dtype or a.shape != b.shape:
+ return False
+ if not a.is_floating_point():
+ return torch.equal(a, b)
+ bits = {1: torch.uint8, 2: torch.int16, 4: torch.int32, 8: torch.int64}[a.element_size()]
+ return torch.equal(a.contiguous().view(bits), b.contiguous().view(bits))
+
+
+def _quantized(name, signs, scale):
+ """Yield the three compressed-tensors parameters for one projection."""
+ rows, cols = signs.shape
+ if scale.shape != (rows, cols // GROUP_SIZE):
+ raise ValueError(f"{name}: scale shape {tuple(scale.shape)} does not match "
+ f"[{rows}, {cols // GROUP_SIZE}]")
+ if scale.dtype != torch.bfloat16:
+ raise ValueError(f"{name}: group scales are {scale.dtype}, not bfloat16")
+ base = name.removesuffix(".weight")
+ yield f"{base}.weight_packed", pack_int4(signs)
+ yield f"{base}.weight_scale", scale.contiguous()
+ yield f"{base}.weight_shape", torch.tensor([rows, cols], dtype=torch.int64)
+
+
+def _from_dense(name, tensor):
+ """A dense projection that must be exactly ternary, as 4-bit parameters."""
+ if tensor.dim() != 2 or tensor.shape[1] % GROUP_SIZE:
+ raise ValueError(f"{name}: shape {tuple(tensor.shape)} cannot be grouped by {GROUP_SIZE}")
+ trits, scale = ternary._to_trits(tensor, GROUP_SIZE)
+ rebuilt = ternary._from_trits(trits, scale, tensor.shape, GROUP_SIZE, tensor.dtype)
+ if not bit_equal(rebuilt, tensor):
+ raise ValueError(f"{name} is not exactly ternary in {GROUP_SIZE}-wide groups; "
+ "refusing to quantize it to W4A16, which would be lossy")
+ signs = (trits.to(torch.int8) - 1).reshape(tensor.shape)
+ yield from _quantized(name, signs, scale)
+
+
+def _iter_packed(path):
+ from safetensors import safe_open
+
+ with safe_open(str(path), framework="pt") as handle:
+ metadata = handle.metadata() or {}
+ if metadata.get("format") != "kittenml-ternary-1.58":
+ raise ValueError(f"not a packed ternary checkpoint: {metadata.get('format')!r}")
+ if int(metadata["group_size"]) != GROUP_SIZE:
+ raise ValueError(f"packed group size {metadata['group_size']} is not {GROUP_SIZE}")
+ packed = json.loads(metadata["packed"])
+ for name in handle.keys():
+ if "::" in name or name.startswith("spk_proj."):
+ continue
+ tensor = handle.get_tensor(name)
+ if is_projection(name):
+ yield from _from_dense(name, tensor)
+ else:
+ yield name, tensor
+ for name, spec in packed.items():
+ if name.startswith("spk_proj."):
+ continue
+ shape = torch.Size(spec["shape"])
+ trits = ternary._unpack_trits(handle.get_tensor(f"{name}::codes"), shape.numel())
+ scale = handle.get_tensor(f"{name}::scale")
+ if not is_projection(name):
+ # Not a layer vLLM quantizes: hand it over dense, exactly as unpacked.
+ yield name, ternary._from_trits(trits, scale, shape, GROUP_SIZE,
+ getattr(torch, spec["dtype"]))
+ continue
+ signs = (trits.to(torch.int8) - 1).reshape(shape)
+ yield from _quantized(name, signs, scale)
+
+
+def _iter_emb4(repo_dir, config):
+ from ..weights import load_lm_weights
+
+ state = load_lm_weights(repo_dir, config, "emb4")
+ for name, tensor in state.items():
+ if name.startswith("spk_proj."):
+ continue
+ if is_projection(name):
+ yield from _from_dense(name, tensor)
+ else:
+ yield name, tensor
+
+
+def _iter_full(lm_dir, shards):
+ from safetensors import safe_open
+
+ if not shards:
+ raise FileNotFoundError(f"no plain model weights in {lm_dir}")
+ for shard in shards:
+ with safe_open(str(shard), framework="pt") as handle:
+ for name in handle.keys():
+ if name.startswith("spk_proj."):
+ continue
+ tensor = handle.get_tensor(name)
+ if is_projection(name):
+ yield from _from_dense(name, tensor)
+ else:
+ yield name, tensor
+
+
+def iter_w4a16(repo_dir, config, variant):
+ """Yield (name, tensor) pairs for vLLM's compressed-tensors W4A16 layers.
+
+ One tensor at a time, so the expanded bf16 state dict is never built. (The
+ bf16 files are memory-mapped, so with ``variant="full"`` the pages read so
+ far still count towards the process's resident memory.)
+
+ - ``variant="packed"`` reads the 1.58-bit file and converts trits directly.
+ - ``variant="full"`` reads the bf16 files and checks every projection is
+ exactly ternary, raising otherwise.
+ - ``variant="emb4"`` unpacks the TL2 body (exactly ternary, checked like
+ ``"full"``) and keeps its 4-bit embedding, which is lossy; only the
+ embedding and the tied output head differ from the bf16 model.
+
+ Projections become ``weight_packed`` (int32), ``weight_scale`` (the stored
+ bf16 scales, unchanged) and ``weight_shape`` (int64 [2]). Other tensors pass
+ through; ``spk_proj.*`` is dropped because it runs outside vLLM.
+ """
+ if variant == "emb4":
+ return _iter_emb4(repo_dir, config)
+ path = weights_path(repo_dir, config, variant)
+ if path is None:
+ return _iter_full(*plain_shards(repo_dir, config))
+ if not path.is_file():
+ raise FileNotFoundError(f"{variant!r} weights not found: {path}")
+ return _iter_packed(path)
diff --git a/kittenml/kittentts2/weights.py b/kittenml/kittentts2/weights.py
index cc52c39..77f2242 100644
--- a/kittenml/kittentts2/weights.py
+++ b/kittenml/kittentts2/weights.py
@@ -1,5 +1,6 @@
"""Read the selected checkpoint identically for Torch and vLLM."""
+import json
from pathlib import Path
@@ -20,8 +21,20 @@ def weights_path(repo_dir, config, name):
return Path(repo_dir) / declared
-def load_lm_weights(repo_dir, config, variant, lm_dir=None):
- """Unpack on CPU without re-quantizing or writing an expanded checkpoint."""
+def plain_shards(repo_dir, config, lm_dir=None):
+ """The bf16 ("full") safetensors files of the language model, in load order."""
+ lm_dir = Path(lm_dir) if lm_dir is not None else Path(repo_dir) / config.get("lm_dir", "lm")
+ return lm_dir, [s for s in sorted(lm_dir.glob("model*.safetensors")) if "-" not in s.stem]
+
+
+def load_lm_weights(repo_dir, config, variant, lm_dir=None, include=None):
+ """Unpack on CPU without re-quantizing or writing an expanded checkpoint.
+
+ `include`, if given, is a predicate on tensor names: only those tensors are
+ read (and, for packed files, unpacked), so callers that need a few tensors
+ do not expand the whole model in host memory. The emb4 reader always
+ decodes the full file and filters afterwards.
+ """
from safetensors import safe_open
from safetensors.torch import load_file
@@ -31,18 +44,30 @@ def load_lm_weights(repo_dir, config, variant, lm_dir=None):
raise FileNotFoundError(f"{variant!r} weights not found: {path}")
if variant == "emb4":
from .tl2_emb4 import load_state_dict
- return load_state_dict(path)
+ state = load_state_dict(path)
+ return state if include is None else {k: v for k, v in state.items() if include(k)}
with safe_open(str(path), framework="pt") as handle:
metadata = handle.metadata() or {}
- tensors = {k: handle.get_tensor(k) for k in handle.keys()}
+ if include is None:
+ tensors = {k: handle.get_tensor(k) for k in handle.keys()}
+ else:
+ tensors = {k: handle.get_tensor(k) for k in handle.keys()
+ if include(k.split("::", 1)[0])}
+ if include is not None and "packed" in metadata:
+ packed = json.loads(metadata["packed"])
+ metadata = {**metadata, "packed": json.dumps(
+ {k: v for k, v in packed.items() if include(k)})}
from .ternary import unpack_state_dict
return unpack_state_dict(tensors, metadata)
- lm_dir = Path(lm_dir) if lm_dir is not None else Path(repo_dir) / config.get("lm_dir", "lm")
+ lm_dir, shards = plain_shards(repo_dir, config, lm_dir)
state = {}
- for shard in sorted(lm_dir.glob("model*.safetensors")):
- if "-" not in shard.stem:
+ for shard in shards:
+ if include is None:
state.update(load_file(str(shard)))
- if not state:
+ continue
+ with safe_open(str(shard), framework="pt") as handle:
+ state.update({k: handle.get_tensor(k) for k in handle.keys() if include(k)})
+ if not shards or (include is None and not state):
raise FileNotFoundError(f"no plain model weights in {lm_dir}")
return state
diff --git a/tests/test_vllm.py b/tests/test_vllm.py
index 2a16ebd..f0744e2 100644
--- a/tests/test_vllm.py
+++ b/tests/test_vllm.py
@@ -199,6 +199,7 @@ def test_engine_receives_quiet_logging_config_without_changing_native_options(mo
return mock.sentinel.engine
with mock.patch("kittenml.kittentts2.vllm.backend.validate_environment"), \
+ mock.patch("torch.cuda.get_device_capability", return_value=(8, 6)), \
mock.patch("transformers.AutoConfig.from_pretrained", return_value=SimpleNamespace(
model_type="qwen3", hidden_size=4)), \
mock.patch("transformers.AutoTokenizer.from_pretrained"), \
@@ -224,6 +225,9 @@ def test_vllm_loader_filters_speaker_weights_and_rejects_missing_parameters():
def named_parameters(self):
return [("model.weight", None)]
+ def named_modules(self):
+ return []
+
model = TinyModel()
state = {"model.weight": torch.ones(2), "spk_proj.0.bias": torch.zeros(2)}
with mock.patch("kittenml.kittentts2.vllm.loader.load_lm_weights", return_value=state):
@@ -292,3 +296,564 @@ def test_sampler_is_neutral_and_speaker_replaces_only_first_embedding():
assert params.extra_args["kittenml"]["sampling"]["temperature"] == 0.7
with pytest.raises(ValueError, match="token budget"):
backend.prepare_request([0, 3, 1], torch.ones(1, 4), settings, 30, advanced)
+
+
+# ── Exact W4A16 (vllm_options={"quantization": "w4a16"}) ───────────────────────
+
+_PROJECTIONS = {"self_attn.q_proj": (256, 256), "self_attn.k_proj": (128, 256),
+ "self_attn.v_proj": (128, 256), "self_attn.o_proj": (256, 256),
+ "mlp.gate_proj": (384, 256), "mlp.up_proj": (384, 256),
+ "mlp.down_proj": (256, 384)}
+
+
+def _ternary(rows, cols, generator):
+ signs = torch.randint(-1, 2, (rows, cols), generator=generator)
+ scale = (torch.rand(rows, cols // 128, generator=generator) * 0.5 + 1e-4).to(torch.bfloat16)
+ scale[0, 0] = 0 # an all-zero group has a zero scale
+ signs[0, :128] = 0
+ return signs.to(torch.bfloat16) * scale.repeat_interleave(128, dim=1)
+
+
+def _w4a16_checkpoint(tmp_path, layers=2, perturb=None):
+ """A small Qwen3-shaped LM written as both full bf16 and packed ternary."""
+ generator = torch.Generator().manual_seed(0)
+ state = {"model.embed_tokens.weight": torch.randn(32, 256, generator=generator).to(torch.bfloat16),
+ "model.norm.weight": torch.rand(256, generator=generator).to(torch.bfloat16),
+ "spk_proj.0.weight": torch.randn(256, 8, generator=generator).to(torch.bfloat16),
+ "spk_proj.0.bias": torch.zeros(256, dtype=torch.bfloat16),
+ "spk_proj.1.weight": torch.ones(256, dtype=torch.bfloat16),
+ "spk_proj.1.bias": torch.zeros(256, dtype=torch.bfloat16)}
+ for layer in range(layers):
+ prefix = f"model.layers.{layer}."
+ state[prefix + "input_layernorm.weight"] = torch.ones(256, dtype=torch.bfloat16)
+ state[prefix + "self_attn.q_norm.weight"] = torch.ones(128, dtype=torch.bfloat16)
+ for name, shape in _PROJECTIONS.items():
+ state[prefix + name + ".weight"] = _ternary(*shape, generator)
+ if perturb is not None:
+ weight = state[perturb]
+ weight[1, 5] = weight[1, :128].abs().max() / 2 # no longer ternary in that group
+ lm = tmp_path / "lm"
+ lm.mkdir()
+ save_file(state, lm / "model.safetensors")
+ tensors, metadata = pack_state_dict(state)
+ save_file(tensors, lm / "model-ternary.safetensors", metadata=metadata)
+ return state, {"lm_packed": "lm/model-ternary.safetensors"}
+
+
+def _unpack_int4(packed, shape):
+ """Inverse of w4a16.pack_int4: int32 words -> int8 values of `shape`."""
+ from kittenml.kittentts2.vllm.w4a16 import NUM_BITS, OFFSET, PACK_FACTOR
+
+ words = packed.to(torch.int64) & 0xFFFFFFFF
+ shifts = torch.arange(PACK_FACTOR, dtype=torch.int64) * NUM_BITS
+ codes = (words.unsqueeze(2) >> shifts) & ((1 << NUM_BITS) - 1)
+ return (codes - OFFSET).reshape(packed.shape[0], -1)[:, :int(shape[1])].to(torch.int8)
+
+
+def _reference_unpack():
+ """compressed-tensors' own unpacker when installed (it ships with vLLM)."""
+ try:
+ from compressed_tensors.compressors.pack_quantized.helpers import unpack_from_int32
+ except ImportError:
+ return _unpack_int4
+ return lambda packed, shape: unpack_from_int32(packed, 4, torch.Size(shape))
+
+
[email protected]("variant", ["packed", "full"])
+def test_w4a16_dequantizes_bit_exactly_to_the_selected_weights(tmp_path, variant):
+ from kittenml.kittentts2.vllm.w4a16 import bit_equal, is_projection, iter_w4a16
+
+ state, config = _w4a16_checkpoint(tmp_path)
+ expected = load_lm_weights(tmp_path, config, variant)
+ produced = dict(iter_w4a16(tmp_path, config, variant))
+ unpack = _reference_unpack()
+ projections = [name for name in state if is_projection(name)]
+ assert len(projections) == 14
+ dense = {name for name in state if not is_projection(name) and not name.startswith("spk_proj.")}
+ assert set(produced) == dense | {name.removesuffix(".weight") + suffix
+ for name in projections
+ for suffix in (".weight_packed", ".weight_scale",
+ ".weight_shape")}
+ for name in dense:
+ assert bit_equal(produced[name], expected[name])
+ for name in projections:
+ base = name.removesuffix(".weight")
+ rows, cols = expected[name].shape
+ packed = produced[base + ".weight_packed"]
+ scale = produced[base + ".weight_scale"]
+ shape = produced[base + ".weight_shape"]
+ assert packed.dtype == torch.int32 and packed.shape == (rows, cols // 8)
+ assert scale.dtype == torch.bfloat16 and scale.shape == (rows, cols // 128)
+ assert shape.dtype == torch.int64 and shape.tolist() == [rows, cols]
+ signs = unpack(packed, shape.tolist())
+ assert set((signs.to(torch.int16) + 8).unique().tolist()) <= {7, 8, 9}
+ weight = signs.to(torch.bfloat16) * scale.repeat_interleave(128, dim=1)
+ assert bit_equal(weight, expected[name]), name
+
+
+def test_w4a16_packed_and_full_sources_give_identical_tensors(tmp_path):
+ from kittenml.kittentts2.vllm.w4a16 import bit_equal, iter_w4a16
+
+ _, config = _w4a16_checkpoint(tmp_path)
+ packed = dict(iter_w4a16(tmp_path, config, "packed"))
+ full = dict(iter_w4a16(tmp_path, config, "full"))
+ assert packed.keys() == full.keys()
+ assert all(bit_equal(packed[k], full[k]) for k in packed)
+
+
+def test_w4a16_int32_bit_order_matches_compressed_tensors():
+ from kittenml.kittentts2.vllm.w4a16 import pack_int4
+
+ # Codes 7,8,9,8,8,8,8,9: column 0 in bits 0-3 ... column 7 in bits 28-31.
+ golden = pack_int4(torch.tensor([[-1, 0, 1, 0, 0, 0, 0, 1]], dtype=torch.int8))
+ assert golden.dtype == torch.int32 and golden.tolist() == [[-1735882361]] # 0x98888987
+ assert pack_int4(torch.zeros(1, 8, dtype=torch.int8)).tolist() == [[0x88888888 - (1 << 32)]]
+ values = torch.randint(-8, 8, (6, 256), generator=torch.Generator().manual_seed(1)).to(torch.int8)
+ assert torch.equal(_unpack_int4(pack_int4(values), values.shape), values)
+ with pytest.raises(ValueError, match="4 bits"):
+ pack_int4(torch.full((1, 8), 8, dtype=torch.int8))
+ helpers = pytest.importorskip("compressed_tensors.compressors.pack_quantized.helpers")
+ assert torch.equal(pack_int4(values), helpers.pack_to_int32(values, 4))
+ assert torch.equal(helpers.unpack_from_int32(pack_int4(values), 4, values.shape), values)
+
+
[email protected]("variant", ["packed", "full"])
+def test_w4a16_refuses_weights_that_are_not_exactly_ternary(tmp_path, variant):
+ from kittenml.kittentts2.vllm.w4a16 import iter_w4a16
+
+ _, config = _w4a16_checkpoint(tmp_path, perturb="model.layers.1.mlp.up_proj.weight")
+ with pytest.raises(ValueError, match="model.layers.1.mlp.up_proj.weight is not exactly ternary"):
+ list(iter_w4a16(tmp_path, config, variant))
+
+
+def test_w4a16_refuses_scales_that_are_not_bfloat16(tmp_path):
+ # Marlin reads the group scales in the activation dtype, which is bfloat16.
+ from kittenml.kittentts2.vllm.w4a16 import iter_w4a16
+
+ weight = torch.zeros(2, 128, dtype=torch.float32)
+ weight[0, :2] = torch.tensor([0.5, -0.5])
+ (tmp_path / "lm").mkdir()
+ save_file({"model.layers.0.self_attn.o_proj.weight": weight}, tmp_path / "lm" / "model.safetensors")
+ with pytest.raises(ValueError, match="o_proj.weight: group scales are torch.float32, not bfloat16"):
+ list(iter_w4a16(tmp_path, {}, "full"))
+
+
+def test_w4a16_accepts_emb4_and_packs_its_ternary_body_exactly(tmp_path):
+ from kittenml.kittentts2.vllm import w4a16
+ from kittenml.kittentts2.vllm.backend import check_options
+
+ signs = torch.randint(-1, 2, (16, 256), dtype=torch.int8)
+ scale = torch.rand(16, 2, dtype=torch.bfloat16) + 0.5
+ weight = (signs.reshape(16, 2, 128).to(torch.float32)
+ * scale.to(torch.float32).unsqueeze(-1)).reshape(16, 256).to(torch.bfloat16)
+ embedding = torch.randn(8, 4, dtype=torch.bfloat16)
+ state = {"model.layers.0.self_attn.o_proj.weight": weight,
+ "model.embed_tokens.weight": embedding,
+ "spk_proj.weight": torch.zeros(2, 2, dtype=torch.bfloat16)}
+ with mock.patch("kittenml.kittentts2.weights.load_lm_weights", return_value=state) as load:
+ out = dict(w4a16.iter_w4a16(tmp_path, {"lm_emb4": "x"}, "emb4"))
+ load.assert_called_once_with(tmp_path, {"lm_emb4": "x"}, "emb4")
+ assert "spk_proj.weight" not in out
+ assert torch.equal(out["model.embed_tokens.weight"], embedding)
+ expected = dict(w4a16._from_dense("model.layers.0.self_attn.o_proj.weight", weight))
+ for key, value in expected.items():
+ assert torch.equal(out[key], value)
+
+ check_options({"quantization": "w4a16"}, "emb4")
+
+
+def test_w4a16_targets_112_fused_modules_for_196_projections():
+ # Qwen3-1.7B shape: 28 layers x 7 ternary projections. vLLM's own Qwen3 model
+ # loads q/k/v into qkv_proj and gate/up into gate_up_proj (its
+ # packed_modules_mapping), so 4 quantized modules per layer.
+ pytest.importorskip("vllm")
+ from vllm.model_executor.models.qwen3 import Qwen3ForCausalLM
+ from kittenml.kittentts2.vllm.w4a16 import is_projection
+
+ fused = {part: module for module, parts in Qwen3ForCausalLM.packed_modules_mapping.items()
+ for part in parts}
+ names = ["model.embed_tokens.weight", "model.norm.weight", "spk_proj.0.weight"]
+ for layer in range(28):
+ names += [f"model.layers.{layer}.{p}.weight" for p in _PROJECTIONS]
+ names += [f"model.layers.{layer}.input_layernorm.weight",
+ f"model.layers.{layer}.self_attn.k_norm.weight"]
+ projections = [name.removesuffix(".weight") for name in names if is_projection(name)]
+ assert len(projections) == 196
+ modules = {name.rsplit(".", 1)[0] + "." + fused.get(name.rsplit(".", 1)[1], name.rsplit(".", 1)[1])
+ for name in projections}
+ assert len(modules) == 112
+ assert {m.rsplit(".", 1)[1] for m in modules} == {"qkv_proj", "o_proj", "gate_up_proj",
+ "down_proj"}
+
+
+def test_vllm_options_validate_quantization_dtype_and_cache_size():
+ from kittenml.kittentts2.vllm.backend import resolve_options
+
+ engine, kitten = resolve_options(None)
+ assert engine == dict(gpu_memory_utilization=0.5, max_model_len=4096, max_num_seqs=8,
+ enforce_eager=False)
+ assert kitten == dict(quantization=None, dtype=None, kv_cache_memory_bytes=None)
+ engine, kitten = resolve_options({"quantization": "w4a16", "dtype": "bfloat16",
+ "kv_cache_memory_bytes": 1 << 30, "max_num_seqs": 4})
+ assert engine["kv_cache_memory_bytes"] == 1 << 30 and engine["max_num_seqs"] == 4
+ assert "quantization" not in engine and "dtype" not in engine
+ assert kitten["quantization"] == "w4a16" and kitten["dtype"] == "bfloat16"
+ for bad, match in [({"quantization": "w8a8"}, "quantization"),
+ ({"quantization": "W4A16"}, "quantization"),
+ ({"dtype": "fp16"}, "dtype"),
+ ({"dtype": "float32"}, "dtype"),
+ ({"kv_cache_memory_bytes": 0}, "kv_cache_memory_bytes"),
+ ({"kv_cache_memory_bytes": 1.5e9}, "kv_cache_memory_bytes"),
+ ({"kv_cache_memory_bytes": True}, "kv_cache_memory_bytes"),
+ ({"kv_cache_dtype": "fp8"}, "unknown vllm_options")]:
+ with pytest.raises(ValueError, match=match):
+ resolve_options(bad)
+
+
+def test_float16_is_refused_with_the_measured_reason():
+ # float16 was measured to overflow at layer 10 with the current checkpoint
+ # (with and without W4A16) and then produce no audio, so it is refused.
+ from kittenml.kittentts2.vllm.backend import resolve_options
+
+ for options in ({"dtype": "float16"}, {"dtype": "float16", "quantization": "w4a16"}):
+ with pytest.raises(ValueError, match=r"'float16' is not supported: this model's residual "
+ r"stream exceeds the float16 range \(measured overflow "
+ r"from layer 10\).*compute capability < 8.0"):
+ resolve_options(options)
+
+
+def test_dtype_follows_hardware_and_gpus_without_bfloat16_are_refused():
+ from kittenml.kittentts2.vllm.backend import validate_environment
+
+ with mock.patch("kittenml.kittentts2.vllm.backend.importlib.util.find_spec", return_value=object()), \
+ mock.patch("torch.cuda.is_available", return_value=True):
+ for capability in ((8, 0), (8, 6), (12, 0)):
+ with mock.patch("torch.cuda.get_device_capability", return_value=capability):
+ validate_environment()
+ for capability, name in (((7, 5), "SM75"), ((7, 0), "SM70")):
+ with mock.patch("torch.cuda.get_device_capability", return_value=capability):
+ with pytest.raises(RuntimeError, match=name + ".*no bfloat16.*float16 range"
+ ".*layer 10.*< 8.0"):
+ validate_environment()
+
+
+def test_vllm_options_are_checked_before_model_download(monkeypatch):
+ resolved = (None, "example/repo", {"type": "KITTEN2"})
+ with mock.patch("kittenml.get_model.resolve_repo", return_value=resolved), \
+ mock.patch("kittenml.kittentts2.vllm.backend.validate_environment"), \
+ mock.patch("kittenml.get_model.snapshot_download") as download:
+ for options, weights, match in [({"quantisation": "w4a16"}, None, "unknown vllm_options"),
+ ({"dtype": "float16"}, None, "float16 range")]:
+ with pytest.raises(ValueError, match=match):
+ KittenTTS("example/repo", backend="vllm", weights=weights, vllm_options=options)
+ monkeypatch.setenv("VLLM_MARLIN_INPUT_DTYPE", "int8")
+ with pytest.raises(RuntimeError, match="W4A8"):
+ KittenTTS("example/repo", backend="vllm", vllm_options={"quantization": "w4a16"})
+ download.assert_not_called()
+
+
+def test_gpus_without_bfloat16_are_refused_before_model_download():
+ resolved = (None, "example/repo", {"type": "KITTEN2"})
+ with mock.patch("kittenml.get_model.resolve_repo", return_value=resolved), \
+ mock.patch("kittenml.kittentts2.vllm.backend.importlib.util.find_spec", return_value=object()), \
+ mock.patch("torch.cuda.is_available", return_value=True), \
+ mock.patch("torch.cuda.get_device_capability", return_value=(7, 5)), \
+ mock.patch("kittenml.get_model.snapshot_download") as download:
+ with pytest.raises(RuntimeError, match="SM75.*no bfloat16"):
+ KittenTTS("example/repo", backend="vllm")
+ download.assert_not_called()
+
+
+def test_w4a16_refuses_marlin_environment_overrides(monkeypatch, tmp_path):
+ from kittenml.kittentts2.vllm.backend import VLLMBackend, check_w4a16_environment
+
+ check_w4a16_environment({})
+ # vLLM reads only VLLM_MARLIN_USE_ATOMIC_ADD=1 as on; other kernels may be disabled.
+ check_w4a16_environment({"VLLM_MARLIN_USE_ATOMIC_ADD": "0",
+ "VLLM_DISABLED_KERNELS": "ExllamaLinearKernel"})
+ for key, value in (("VLLM_MARLIN_INPUT_DTYPE", "int8"), ("VLLM_MARLIN_INPUT_DTYPE", "fp8"),
+ ("VLLM_MARLIN_USE_ATOMIC_ADD", "1"),
+ ("VLLM_DISABLED_KERNELS", "ConchLinearKernel,MarlinLinearKernel")):
+ with pytest.raises(RuntimeError, match=key):
+ check_w4a16_environment({key: value})
+ monkeypatch.setenv("VLLM_MARLIN_INPUT_DTYPE", "int8")
+ owner = SimpleNamespace(device="cuda", repo_dir=tmp_path, config={}, weights="packed",
+ _load_lm_weights=mock.Mock())
+ with mock.patch("kittenml.kittentts2.vllm.backend.validate_environment"):
+ with pytest.raises(RuntimeError, match="W4A8"):
+ VLLMBackend(owner, {"quantization": "w4a16"})
+ owner._load_lm_weights.assert_not_called()
+
+
+def test_w4a16_hf_overrides_payload():
+ from kittenml.kittentts2.vllm.backend import quantization_args
+ from kittenml.kittentts2.vllm.w4a16 import QUANTIZATION_CONFIG
+
+ assert quantization_args(None) == {}
+ args = quantization_args("w4a16")
+ assert args["quantization"] == "compressed-tensors"
+ payload = args["hf_overrides"]["quantization_config"]
+ assert payload == {
+ "quant_method": "compressed-tensors", "format": "pack-quantized",
+ "config_groups": {"group_0": {
+ "targets": ["Linear"],
+ "weights": {"num_bits": 4, "type": "int", "symmetric": True, "strategy": "group",
+ "group_size": 128, "dynamic": False, "actorder": None},
+ "input_activations": None}},
+ "ignore": ["lm_head"]}
+ payload["total_num_heads"] = 16 # vLLM writes into the dict it receives
+ assert "total_num_heads" not in QUANTIZATION_CONFIG
+
+
+def test_w4a16_payload_selects_vllm_uint4b8_weight_only_scheme():
+ pytest.importorskip("vllm")
+ from vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors import (
+ CompressedTensorsConfig)
+ from vllm.model_executor.layers.quantization.compressed_tensors.schemes.compressed_tensors_wNa16 import (
+ WNA16_SUPPORTED_TYPES_MAP)
+ from kittenml.kittentts2.vllm.w4a16 import OFFSET, quantization_config
+
+ config = CompressedTensorsConfig.from_config(quantization_config())
+ assert config.ignore == ["lm_head"] and config.quant_format == "pack-quantized"
+ scheme = config.target_scheme_map["Linear"]
+ assert scheme["input_activations"] is None
+ assert CompressedTensorsConfig._is_wNa16_group_channel(scheme["weights"], None)
+ weights = scheme["weights"]
+ assert (weights.num_bits, weights.group_size, weights.symmetric) == (4, 128, True)
+ assert WNA16_SUPPORTED_TYPES_MAP[4].bias == OFFSET == 8 # uint4b8: code = sign + 8
+
+
+def test_engine_receives_w4a16_arguments_and_main_process_loads_only_conditioning(tmp_path):
+ pytest.importorskip("vllm")
+ from kittenml.kittentts2.vllm.backend import VLLMBackend
+
+ import os
+ seen = []
+
+ def create_engine(**kwargs):
+ seen.append(dict(kwargs, disabled_kernels=os.environ.get("VLLM_DISABLED_KERNELS")))
+
+ def run(options):
+ owner = SimpleNamespace(device="cuda", repo_dir=tmp_path, config={}, weights="packed",
+ token_map=None, _load_lm_weights=mock.Mock(return_value={
+ "model.embed_tokens.weight": torch.zeros(2, 4)}))
+ with mock.patch("kittenml.kittentts2.vllm.backend.validate_environment"), \
+ mock.patch("torch.cuda.get_device_capability", return_value=(8, 6)), \
+ mock.patch("transformers.AutoConfig.from_pretrained", return_value=SimpleNamespace(
+ model_type="qwen3", hidden_size=4)), \
+ mock.patch("transformers.AutoTokenizer.from_pretrained"), \
+ mock.patch("kittenml.kittentts2.speaker.load_speaker_head"), \
+ mock.patch("vllm.LLM", side_effect=create_engine):
+ VLLMBackend(owner, options)
+ include = owner._load_lm_weights.call_args.kwargs["include"]
+ assert include("model.embed_tokens.weight") and include("spk_proj.0.weight")
+ assert not include("model.layers.0.self_attn.q_proj.weight")
+ return owner, seen[-1]
+
+ os.environ.pop("VLLM_DISABLED_KERNELS", None)
+ owner, kwargs = run(None)
+ assert {"quantization", "hf_overrides", "kv_cache_memory_bytes"}.isdisjoint(kwargs)
+ assert "quantization" not in kwargs["model_loader_extra_config"]
+ assert kwargs["dtype"] is torch.bfloat16 and owner.dtype is torch.bfloat16
+ assert kwargs["disabled_kernels"] is None
+ owner, kwargs = run({"quantization": "w4a16"})
+ # Machete, which vLLM prefers on Hopper, is disabled for the engine only.
+ assert kwargs["disabled_kernels"] == "MacheteLinearKernel"
+ assert "VLLM_DISABLED_KERNELS" not in os.environ
+ owner, kwargs = run({"quantization": "w4a16", "kv_cache_memory_bytes": 1 << 30,
+ "dtype": "bfloat16"})
+ assert kwargs["quantization"] == "compressed-tensors"
+ assert kwargs["hf_overrides"]["quantization_config"]["ignore"] == ["lm_head"]
+ assert kwargs["model_loader_extra_config"]["quantization"] == "w4a16"
+ assert kwargs["kv_cache_memory_bytes"] == 1 << 30
+ assert kwargs["dtype"] is torch.bfloat16 and owner.dtype is torch.bfloat16
+
+
+def _linear_with_kernel(kind, act_type=torch.bfloat16):
+ """A module carrying a compressed-tensors scheme whose kernel is `kind`."""
+ kernel = object.__new__(kind)
+ kernel.config = SimpleNamespace(act_type=act_type)
+ module = torch.nn.Module()
+ module.scheme = SimpleNamespace(kernel=kernel)
+ return module
+
+
+def test_vllm_loader_streams_w4a16_tensors_without_speaker_head():
+ pytest.importorskip("vllm")
+ from vllm.model_executor.kernels.linear import MarlinLinearKernel
+ from kittenml.kittentts2.vllm.loader import KittenModelLoader
+
+ loader = object.__new__(KittenModelLoader)
+ loader.load_config = SimpleNamespace(model_loader_extra_config={
+ "repo_dir": "unused", "config": {}, "weights": "packed", "quantization": "w4a16"})
+
+ class TinyModel:
+ def load_weights(self, weights):
+ self.loaded = dict(weights)
+ return set(self.loaded)
+
+ def named_parameters(self):
+ return [("model.layers.0.mlp.down_proj.weight_packed", None)]
+
+ def named_modules(self):
+ return [("model.layers.0.mlp.down_proj", _linear_with_kernel(MarlinLinearKernel))]
+
+ model = TinyModel()
+ tensors = [("model.layers.0.mlp.down_proj.weight_packed", torch.zeros(1, 1, dtype=torch.int32))]
+ with mock.patch("kittenml.kittentts2.vllm.w4a16.iter_w4a16",
+ return_value=iter(tensors)) as convert:
+ loader.load_weights(model, SimpleNamespace(dtype=torch.bfloat16))
+ convert.assert_called_once_with("unused", {}, "packed")
+ assert set(model.loaded) == {"model.layers.0.mlp.down_proj.weight_packed"}
+ loader.load_config.model_loader_extra_config["quantization"] = "int8"
+ with pytest.raises(ValueError, match="unknown KittenTTS quantization"):
+ loader.load_weights(model, None)
+
+
+def test_vllm_loader_allows_only_kv_cache_scales_to_be_missing():
+ # compressed-tensors adds KV-cache scale parameters to every attention layer;
+ # the checkpoint has none (the KV cache stays in the model dtype). On a GPU,
+ # the strict missing-weights check rejected the real model because of them.
+ pytest.importorskip("vllm")
+ from vllm.model_executor.kernels.linear import MarlinLinearKernel
+ from vllm.model_executor.layers.quantization.kv_cache import BaseKVCacheMethod
+ from kittenml.kittentts2.vllm.loader import KittenModelLoader
+
+ loader = object.__new__(KittenModelLoader)
+ loader.load_config = SimpleNamespace(model_loader_extra_config={
+ "repo_dir": "unused", "config": {}, "weights": "packed", "quantization": "w4a16"})
+ attn = torch.nn.Module()
+ attn.quant_method = object.__new__(BaseKVCacheMethod)
+ attn.k_scale = torch.nn.Parameter(torch.tensor(-1.0), requires_grad=False)
+ attn.k_zero_point = torch.nn.Parameter(torch.tensor(0), requires_grad=False)
+ model = torch.nn.Module()
+ model.attn = attn
+ model.proj = torch.nn.Linear(2, 2, bias=False)
+ model.proj.scheme = _linear_with_kernel(MarlinLinearKernel).scheme
+ model.load_weights = lambda weights: {name for name, _ in weights}
+ with mock.patch("kittenml.kittentts2.vllm.w4a16.iter_w4a16",
+ return_value=iter([("proj.weight", torch.zeros(2, 2))])):
+ loader.load_weights(model, SimpleNamespace(dtype=torch.bfloat16))
+ with mock.patch("kittenml.kittentts2.vllm.w4a16.iter_w4a16", return_value=iter([])):
+ with pytest.raises(ValueError, match=r"missing from the checkpoint: \['proj.weight'\]"):
+ loader.load_weights(model, SimpleNamespace(dtype=torch.bfloat16))
+
+
[email protected]("variant", ["packed", "full"])
+def test_selected_tensor_loading_matches_full_load(tmp_path, variant):
+ from kittenml.kittentts2.vllm.backend import _main_process_tensor
+
+ _, config = _w4a16_checkpoint(tmp_path, layers=1)
+ everything = load_lm_weights(tmp_path, config, variant)
+ selected = load_lm_weights(tmp_path, config, variant, include=_main_process_tensor)
+ assert set(selected) == {"model.embed_tokens.weight", "spk_proj.0.weight", "spk_proj.0.bias",
+ "spk_proj.1.weight", "spk_proj.1.bias"}
+ assert all(torch.equal(selected[k], everything[k]) for k in selected)
+
+
+def test_vllm_loader_refuses_w4a16_layers_on_any_kernel_but_marlin():
+ # On Hopper vLLM would pick Machete, and with VLLM_DISABLED_KERNELS it can fall
+ # back to Triton or others; exact output was verified on Marlin only.
+ pytest.importorskip("vllm")
+ from vllm.model_executor.kernels.linear import MarlinLinearKernel
+ from vllm.model_executor.kernels.linear.mixed_precision.machete import MacheteLinearKernel
+ from kittenml.kittentts2.vllm.loader import KittenModelLoader
+
+ loader = object.__new__(KittenModelLoader)
+ loader.load_config = SimpleNamespace(model_loader_extra_config={
+ "repo_dir": "unused", "config": {}, "weights": "packed", "quantization": "w4a16"})
+ model = torch.nn.Module()
+ model.a = _linear_with_kernel(MarlinLinearKernel)
+ model.b = _linear_with_kernel(MacheteLinearKernel)
+ model.load_weights = mock.Mock(return_value=set())
+ with mock.patch("kittenml.kittentts2.vllm.w4a16.iter_w4a16") as convert:
+ with pytest.raises(RuntimeError, match=r"\['MacheteLinearKernel'\] for 1 linear layers"):
+ loader.load_weights(model, SimpleNamespace(dtype=torch.bfloat16))
+ with pytest.raises(RuntimeError, match="no quantized linear layers"):
+ loader.load_weights(torch.nn.Module(), SimpleNamespace(dtype=torch.bfloat16))
+ convert.assert_not_called()
+ model.load_weights.assert_not_called()
+
+
+def test_vllm_loader_refuses_w4a8_activations():
+ # VLLM_MARLIN_INPUT_DTYPE keeps MarlinLinearKernel but sets its activation type
+ # to int8/fp8 (W4A8). The loader runs in the engine process and checks the
+ # activation type vLLM gave each layer.
+ pytest.importorskip("vllm")
+ from vllm.model_executor.kernels.linear import MarlinLinearKernel
+ from kittenml.kittentts2.vllm.loader import KittenModelLoader
+
+ loader = object.__new__(KittenModelLoader)
+ loader.load_config = SimpleNamespace(model_loader_extra_config={
+ "repo_dir": "unused", "config": {}, "weights": "packed", "quantization": "w4a16"})
+ bf16 = SimpleNamespace(dtype=torch.bfloat16)
+ model = torch.nn.Module()
+ model.a = _linear_with_kernel(MarlinLinearKernel)
+ model.b = _linear_with_kernel(MarlinLinearKernel, act_type=torch.int8)
+ model.load_weights = mock.Mock(return_value=set())
+ with mock.patch("kittenml.kittentts2.vllm.w4a16.iter_w4a16") as convert:
+ with pytest.raises(RuntimeError, match=r"1 linear layers would quantize their "
+ r"activations to \['torch.int8'\].*"
+ r"VLLM_MARLIN_INPUT_DTYPE"):
+ loader.load_weights(model, bf16)
+ model.b = _linear_with_kernel(MarlinLinearKernel)
+ with pytest.raises(RuntimeError, match="engine dtype is torch.float16"):
+ loader.load_weights(model, SimpleNamespace(dtype=torch.float16))
+ convert.assert_not_called()
+ model.load_weights.assert_not_called()
+
+
+def test_engine_sees_the_checked_environment_after_an_earlier_engine(monkeypatch, tmp_path):
+ # In single-process mode vLLM freezes its view of the environment once an
+ # engine has started (enable_envs_cache). A later W4A16 engine must see the
+ # environment KittenTTS checked, not the frozen one.
+ pytest.importorskip("vllm")
+ import vllm.envs
+ from kittenml.kittentts2.vllm.backend import VLLMBackend
+
+ seen = []
+
+ def create_engine(**kwargs):
+ seen.append((vllm.envs.VLLM_MARLIN_INPUT_DTYPE, vllm.envs.VLLM_DISABLED_KERNELS))
+
+ owner = SimpleNamespace(device="cuda", repo_dir=tmp_path, config={}, weights="packed",
+ token_map=None, _load_lm_weights=mock.Mock(return_value={
+ "model.embed_tokens.weight": torch.zeros(2, 4)}))
+ monkeypatch.delenv("VLLM_DISABLED_KERNELS", raising=False)
+ monkeypatch.setenv("VLLM_MARLIN_INPUT_DTYPE", "int8")
+ try:
+ vllm.envs.enable_envs_cache() # as after a first engine in this process
+ monkeypatch.delenv("VLLM_MARLIN_INPUT_DTYPE")
+ assert vllm.envs.VLLM_MARLIN_INPUT_DTYPE == "int8"
+ with mock.patch("kittenml.kittentts2.vllm.backend.validate_environment"), \
+ mock.patch("torch.cuda.get_device_capability", return_value=(8, 6)), \
+ mock.patch("transformers.AutoConfig.from_pretrained", return_value=SimpleNamespace(
+ model_type="qwen3", hidden_size=4)), \
+ mock.patch("transformers.AutoTokenizer.from_pretrained"), \
+ mock.patch("kittenml.kittentts2.speaker.load_speaker_head"), \
+ mock.patch("vllm.LLM", side_effect=create_engine):
+ VLLMBackend(owner, {"quantization": "w4a16"})
+ finally:
+ vllm.envs.disable_envs_cache()
+ assert seen == [(None, ["MacheteLinearKernel"])]
+
+
+def test_w4a16_exactness_checks_compare_bits_not_values(tmp_path):
+ # torch.equal treats -0.0 and +0.0 as equal; the W4A16 checks must not.
+ from kittenml.kittentts2.vllm.w4a16 import bit_equal, iter_w4a16
+
+ zero = torch.zeros(3, dtype=torch.bfloat16)
+ assert torch.equal(-zero, zero) and not bit_equal(-zero, zero)
+ assert bit_equal(zero, zero.clone()) and not bit_equal(zero, zero.float())
+ weight = torch.zeros(2, 128, dtype=torch.bfloat16)
+ weight[0, :2] = torch.tensor([0.5, -0.5])
+ weight[1, 3] = -0.0 # a ternary rebuild gives +0.0 here
+ (tmp_path / "lm").mkdir()
+ save_file({"model.layers.0.self_attn.o_proj.weight": weight},
+ tmp_path / "lm" / "model.safetensors")
+ with pytest.raises(ValueError, match="not exactly ternary"):
+ list(iter_w4a16(tmp_path, {}, "full"))
+ weight[1, 3] = 0.0
+ save_file({"model.layers.0.self_attn.o_proj.weight": weight},
+ tmp_path / "lm" / "model.safetensors")
+ assert len(list(iter_w4a16(tmp_path, {}, "full"))) == 3