Open-source State-of-the-art TTS model which runs on a CPU π»
Python
Add optional vLLM acceleration for KittenTTS 2
haaziq-sl471f8430724b
diff --git a/README.md b/README.mdindex 27e377b..6453233 100644--- a/README.md+++ b/README.md@@ -32,6 +32,7 @@ The library also ships the original [**lightweight legacy models**](docs/onnx-mo- [Voice cloning](#voice-cloning)- [Expression controls](#expression-controls)- [Running on CPU](#running-on-cpu)+- [Running with vLLM](#running-with-vllm)- [Long text and streaming](#long-text-and-streaming)- [Documentation](#documentation)- [System Requirements](#system-requirements)@@ -185,6 +186,32 @@ KittenTTS 2 runs on CPU out of the box β `device` is auto-detected β but the[kitten-tts-2-cpp](https://github.com/KittenML/kitten-tts-2-cpp), our llama.cpp fork. It readsthe GGUF weights in the model repository's `cpp/` directory.+## Running with vLLM++For GPU inference with vLLM on Linux and an NVIDIA CUDA GPU, install the optional backend:++```bash+pip install "kittenml[vllm]"+```++Select `backend="vllm"`:++```python+from kittenml import KittenTTS++def main():+ m = KittenTTS("KittenML/kitten-tts-2", backend="vllm")+ m.generate_to_file("Hello there.", "output.wav", voice="Bruno")+ print("Saved output.wav")++if __name__ == "__main__":+ main()+```++Keep the main guard when running a script: vLLM starts a worker process. The same+voices, decoders, weight choices and generation arguments apply. See the+[vLLM guide](docs/vllm.md) for memory settings and benchmarks.+## Long text and streamingdiff --git a/docs/README.md b/docs/README.mdindex a575890..33d92ff 100644--- a/docs/README.md+++ b/docs/README.md@@ -7,6 +7,7 @@ Fuller detail than the [project README](../README.md), which covers getting star| [api.md](api.md) | The KittenTTS 2 API: every argument to `generate`, streaming, and the advanced knobs || [voices-and-expression.md](voices-and-expression.md) | The 47 built-in voices, the emotion and vocal-event tags, the ten languages || [decoders.md](decoders.md) | How audio is decoded, and the smaller quantised decoders |+| [vllm.md](vllm.md) | Optional NVIDIA GPU inference with vLLM || [text-normalization.md](text-normalization.md) | How written text becomes spoken text || [architecture.md](architecture.md) | What the model is, package layout, vendored components || [onnx-models.md](onnx-models.md) | The lightweight CPU models and their API |diff --git a/docs/api.md b/docs/api.mdindex a02dd5c..63404f9 100644--- a/docs/api.md+++ b/docs/api.md@@ -14,6 +14,8 @@ Loads whichever model family the repository's `config.json` declares.| `hf_token` | `str` | `None` | Token, only for downloading a private model repository || `decoder` | `str` | repo default | Which decoder to load; see [decoders.md](decoders.md) || `weights` | `str` | `"packed"` | Which weight packing to load: `"packed"` (lossless), `"emb4"` (smaller, quantised embedding), `"full"` (bf16) |+| `backend` | `str` | `None` | Default PyTorch inference, or `"vllm"` for optional NVIDIA GPU inference on Linux |+| `vllm_options` | `dict` | `None` | Engine memory and context settings; see [vLLM](vllm.md) |# `model.generate(text, voice=None, reference=None, ...)`diff --git a/docs/vllm.md b/docs/vllm.mdnew file mode 100644index 0000000..b0d32c6--- /dev/null+++ b/docs/vllm.md@@ -0,0 +1,72 @@+# Running with vLLM++The optional `vllm` backend runs KittenTTS 2's language model on an NVIDIA CUDA+GPU. Audio decoding, voice cloning and text preparation use the existing+KittenTTS implementations. The default backend remains PyTorch and already+supports CUDA.++Install the optional backend and select `backend="vllm"` as shown in the+[README](../README.md#running-with-vllm). This backend requires Linux and uses+vLLM 0.31.0; see its [GPU requirements](https://docs.vllm.ai/en/v0.31.0/getting_started/installation/gpu/)+for compatible NVIDIA drivers.+It loads the selected published weights directly; no manual export+or additional language-model checkpoint is required. Packed weights are expanded in+memory, so their smaller download size does not reduce GPU memory use.++The engine defaults to a 4,096-token context, up to eight active sequences, and+half the GPU memory, leaving space for audio decoding. In the README example,+pass `vllm_options` when constructing the model to change these settings:++```python+m = KittenTTS("KittenML/kitten-tts-2", backend="vllm",+ vllm_options={"gpu_memory_utilization": 0.5,+ "max_model_len": 4096})+```++`gpu_memory_utilization` is a fraction of the GPU's total VRAM. It sets vLLM's+budget; the audio decoder and other GPU processes need additional memory.++If VRAM is limited, reduce `max_model_len` to lower cache requirements. If startup+reports no available memory for cache blocks, increase `gpu_memory_utilization`+to give vLLM a larger share of VRAM, leaving space for audio decoding and other+GPU processes.++Routine engine logs are quiet by default; warnings and errors remain visible.+Set `VLLM_LOGGING_LEVEL=INFO` before starting Python to see startup diagnostics.++The context must fit the reference prompt and the requested generation budget.+Long text still uses KittenTTS's sentence chunking. `generate_stream` yields+completed audio chunks, just like the default backend.++`vllm_options` also accepts `max_num_seqs` and `enforce_eager`. Select a GPU with+`CUDA_VISIBLE_DEVICES` before starting Python. Sampling implementations differ+between the backends, so a matching seed does not guarantee identical audio.++## Benchmarks++Measured on an RTX 3060 (12 GB), Linux, Python 3.12 and NVIDIA driver 580.95.05,+using `weights="packed"`, the default decoder and the stable preset. These are+median end-to-end synthesis times from five runs after two warmups per voice.+Short, medium and long inputs contain 61, 226 and 680 characters; the long input+is split into two chunks. Model initialization and file writes are excluded.++| Input / voice | PyTorch GPU | vLLM | Speedup |+|---|---:|---:|---:|+| Short / Bruno | 6.01 s | 2.05 s | 2.93x |+| Short / Bella | 6.98 s | 2.54 s | 2.75x |+| Medium / Bruno | 22.93 s | 5.49 s | 4.18x |+| Medium / Bella | 27.95 s | 8.26 s | 3.39x |+| Long / Bruno | 56.77 s | 17.56 s | 3.23x |+| Long / Bella | 79.24 s | 24.04 s | 3.30x |++The PyTorch column uses unchanged upstream code in a fresh normal installation+(PyTorch 2.14.1, Transformers 5.18.0). With versions matched to the vLLM install+(PyTorch 2.13.0, Transformers 5.17.0, vLLM 0.31.0), the measured speedup was+3.05-4.49x. Generated audio lengths differ between samplers; vLLM's synthesis+time was 0.44-0.48 seconds per second of output audio in these tests.++vLLM used about 8.5 GiB of GPU memory versus 5.8 GiB for the normal installation.+Cloning without `reference_text` also loads Whisper; those tests used up to+11.5 GiB. Supplying the transcript skips that model.+Initialization with cached downloads took 51 seconds versus 10 seconds; the+first vLLM startup took about 162 seconds. Reuse the loaded model across calls.diff --git a/kittenml/get_model.py b/kittenml/get_model.pyindex 54c6188..b76afa1 100644--- a/kittenml/get_model.py+++ b/kittenml/get_model.py@@ -63,25 +63,35 @@ class KittenTTS(KittenTTSOnnx):"""def __new__(cls, model_name="KittenML/kitten-tts-nano-0.8", cache_dir=None,- backend=None, device=None, hf_token=None, decoder=None, weights=None):+ backend=None, device=None, hf_token=None, decoder=None, weights=None,+ vllm_options=None):local_dir, repo_id, config = resolve_repo(model_name, cache_dir)model_type = config.get("type")if model_type == KITTEN2_TYPE:from .kittentts2 import KittenTTS2+ if backend == "vllm":+ from .kittentts2.vllm.backend import validate_environment+ validate_environment(device)+ elif vllm_options is not None:+ raise ValueError("vllm_options requires backend='vllm'")if local_dir is None:local_dir = snapshot_download(repo_id=repo_id, cache_dir=cache_dir, token=hf_token,allow_patterns=_kitten2_files(config, decoder, weights))# Returning a different class means __init__ below is never called.return KittenTTS2(local_dir, config, device=device, cache_dir=cache_dir,- hf_token=hf_token, decoder=decoder, weights=weights)+ hf_token=hf_token, decoder=decoder, weights=weights,+ backend=backend, vllm_options=vllm_options)if model_type not in ONNX_TYPES:raise ValueError(f"Unsupported model type {model_type!r}; expected one of "f"{[*ONNX_TYPES, KITTEN2_TYPE]}")+ if backend == "vllm" or vllm_options is not None:+ raise ValueError("the vLLM backend requires a KittenTTS 2 model")+instance = super().__new__(cls)# Hand the already-resolved repository to KittenTTSOnnx.__init__ rather# than making it hit the Hub a second time.diff --git a/kittenml/kittentts2/model.py b/kittenml/kittentts2/model.pyindex 60db654..8d356ee 100644--- a/kittenml/kittentts2/model.py+++ b/kittenml/kittentts2/model.py@@ -92,7 +92,7 @@ class KittenTTS2:"""def __init__(self, repo_dir, config, device=None, cache_dir=None, hf_token=None,- decoder=None, weights=None):+ decoder=None, weights=None, backend=None, vllm_options=None):self.repo_dir = pathlib.Path(repo_dir)self.config = configself.cache_dir = cache_dir@@ -112,7 +112,12 @@ class KittenTTS2:self.chunk_min_chars = generation.get("chunk_min_chars", text_utils.CHUNK_MIN_CHARS)self.weights = weights or config.get("default_weights", "packed")- self._load_language_model()+ self.backend = "vllm" if backend == "vllm" else "torch"+ if self.backend == "vllm":+ from .vllm.backend import VLLMBackend+ self.model = VLLMBackend(self, vllm_options)+ else:+ self._load_language_model()self.decoders = config.get("decoders", {})self.decoder = decoder or config.get("default_decoder", "default")self.codec = S3Codec(device=self.device, cache_dir=cache_dir,@@ -233,26 +238,15 @@ class KittenTTS2:return modeldef _weights_path(self, name):- """Where a named weight variant lives, or None to read the plain files."""- if name == "full":- return None- key = {"packed": "lm_packed", "emb4": "lm_emb4"}.get(name)- if key is None:- raise ValueError(f"Unknown weights {name!r}. "- f"Choose from: {self.available_weights}")- declared = self.config.get(key)- if not declared:- raise ValueError(f"this repository declares no {name!r} weights")- return self.repo_dir / declared+ """Where a named weight variant lives, or None for the plain files."""+ from .weights import weights_path+ return weights_path(self.repo_dir, self.config, name)@propertydef available_weights(self):"""Weight variants this repository offers."""- names = ["full"]- for name, key in (("packed", "lm_packed"), ("emb4", "lm_emb4")):- if self.config.get(key):- names.append(name)- return names+ from .weights import available_weights+ return available_weights(self.config)def _load_lm_weights(self, lm_dir):"""The language model's weights, in whichever packing was selected.@@ -262,30 +256,8 @@ class KittenTTS2:"emb4" is smaller again but quantises the embedding, which is lossy βsee tl2_emb4.py. "full" reads the plain bf16 files."""- from safetensors import safe_open- from safetensors.torch import load_file-- path = self._weights_path(self.weights)- if path is not None:- if not path.is_file():- raise FileNotFoundError(f"{self.weights!r} weights not found: {path}")- if self.weights == "emb4":- from .tl2_emb4 import load_state_dict- return load_state_dict(path)- with safe_open(str(path), framework="pt") as handle:- metadata = handle.metadata() or {}- tensors = {k: handle.get_tensor(k) for k in handle.keys()}- from .ternary import unpack_state_dict- return unpack_state_dict(tensors, metadata)-- state = {}- for shard in sorted(pathlib.Path(lm_dir).glob("model*.safetensors")):- if "-" in shard.stem: # a packed variant, not a plain shard- continue- state.update(load_file(str(shard)))- if not state:- raise FileNotFoundError(f"no plain model weights in {lm_dir}")- return state+ from .weights import load_lm_weights+ return load_lm_weights(self.repo_dir, self.config, self.weights, lm_dir=lm_dir)def _load_voices(self):voices_path = self.repo_dir / self.config.get("voices", "voices/voices.json")@@ -385,6 +357,10 @@ class KittenTTS2:reference_prefix=reference["prefix"] if use_reference_prompt else None,emotion_ids=emotion_ids)+ if self.backend == "vllm":+ return self.model.generate_tokens(prompt, reference["embedding"], settings,+ max_new_tokens, advanced)+processors = build_logits_processors(self.token_map,temperature=settings["temperature"], top_k=settings["top_k"],diff --git a/kittenml/kittentts2/speaker.py b/kittenml/kittentts2/speaker.pyindex 9abac2f..dff8756 100644--- a/kittenml/kittentts2/speaker.py+++ b/kittenml/kittentts2/speaker.py@@ -24,19 +24,11 @@ from .speaker_embedding import SAMPLE_RATEWHISPER_MODEL = "openai/whisper-large-v3"-def attach_speaker_head(model, checkpoint_dir, device, dtype,- spk_dim=SPEAKER_EMBEDDING_DIM, state=None):- """Restore `model.spk_proj` from the checkpoint and wire up its prefill hook.-- Without this the model runs unconditioned β it still produces speech, but the- voice is arbitrary, so a missing head is an error rather than a warning.-- `state` lets a caller pass weights it has already read, which avoids loading- several gigabytes twice just to find four tensors.- """- hidden = int(model.config.hidden_size)- model.spk_proj = nn.Sequential(nn.Linear(spk_dim, hidden, bias=True),- nn.LayerNorm(hidden)).to(dtype=dtype)+def load_speaker_head(hidden, checkpoint_dir, device, dtype,+ spk_dim=SPEAKER_EMBEDDING_DIM, state=None):+ """Restore the trained projection without constructing a language model."""+ head = nn.Sequential(nn.Linear(spk_dim, int(hidden), bias=True),+ nn.LayerNorm(int(hidden))).to(dtype=dtype)if state is None:state = {}@@ -48,8 +40,19 @@ def attach_speaker_head(model, checkpoint_dir, device, dtype,raise ValueError(f"no spk_proj.* weights in {checkpoint_dir} β this checkpoint cannot be ""speaker-conditioned, and generating from it would ignore the voice")- model.spk_proj.load_state_dict(head_state, strict=True)- model.spk_proj.to(device=device, dtype=dtype).eval()+ head.load_state_dict(head_state, strict=True)+ return head.to(device=device, dtype=dtype).eval()+++def attach_speaker_head(model, checkpoint_dir, device, dtype,+ spk_dim=SPEAKER_EMBEDDING_DIM, state=None):+ """Restore `model.spk_proj` and inject it at prefill.++ A missing head is an error: speech would otherwise ignore the chosen voice.+ Pass an already-loaded `state` to avoid reading the checkpoint twice.+ """+ model.spk_proj = load_speaker_head(model.config.hidden_size, checkpoint_dir,+ device, dtype, spk_dim=spk_dim, state=state)def _inject_speaker(module, args, kwargs):embedding = kwargs.pop("speaker_embedding", None)diff --git a/kittenml/kittentts2/tl2_emb4.py b/kittenml/kittentts2/tl2_emb4.pyindex ccbb419..9c31aea 100644--- a/kittenml/kittentts2/tl2_emb4.py+++ b/kittenml/kittentts2/tl2_emb4.py@@ -87,7 +87,7 @@ def unpack_tl2(buf, out_features, in_features):def _hadamard(n):- matrix = torch.tensor([[1.0]])+ matrix = torch.tensor([[1.0]], dtype=torch.float32)for _ in range(int(round(math.log2(n)))):matrix = torch.cat([torch.cat([matrix, matrix], 1),torch.cat([matrix, -matrix], 1)], 0)diff --git a/kittenml/kittentts2/vllm/__init__.py b/kittenml/kittentts2/vllm/__init__.pynew file mode 100644index 0000000..780a557--- /dev/null+++ b/kittenml/kittentts2/vllm/__init__.py@@ -0,0 +1 @@+"""Optional vLLM integration for KittenTTS 2."""diff --git a/kittenml/kittentts2/vllm/backend.py b/kittenml/kittentts2/vllm/backend.pynew file mode 100644index 0000000..25deebb--- /dev/null+++ b/kittenml/kittentts2/vllm/backend.py@@ -0,0 +1,116 @@+"""Optional GPU language-model backend; waveform decoding stays in KittenTTS."""++from dataclasses import asdict+import contextlib+import importlib.util+import os+import sys++import torch+++def _engine_environment():+ # KittenTTS already applies its warpers. Use vLLM's native sampler by default+ # so a pip installation needs no nvcc for FlashInfer's sampling JIT. The+ # worker inherits these settings, including quiet startup defaults. Restore+ # the caller's environment afterwards and respect explicit overrides.+ defaults = {"VLLM_USE_FLASHINFER_SAMPLER": "0",+ "VLLM_LOGGING_LEVEL": "WARNING", "TQDM_DISABLE": "1"}+ previous = {key: os.environ.get(key) for key in defaults}+ for key, value in defaults.items():+ os.environ.setdefault(key, value)+ try:+ yield+ finally:+ for key, value in previous.items():+ if value is None:+ os.environ.pop(key, None)+ else:+ os.environ[key] = value+++def validate_environment(device=None):+ if sys.platform != "linux":+ raise RuntimeError("the vLLM backend requires Linux and an NVIDIA CUDA GPU")+ if importlib.util.find_spec("vllm") is None:+ raise ImportError('install the optional backend with pip install "kittenml[vllm]"')+ selected = torch.device(device or "cuda")+ if selected.type != "cuda" or not torch.cuda.is_available():+ raise RuntimeError("the vLLM backend requires an NVIDIA CUDA GPU")+ if selected.index not in (None, 0):+ raise ValueError("select the GPU with CUDA_VISIBLE_DEVICES and use device='cuda'")+++class VLLMBackend:+ def __init__(self, owner, options=None):+ validate_environment(owner.device)+ from transformers import AutoConfig, AutoTokenizer+ from ..speaker import load_speaker_head++ options = dict(options or {})+ defaults = dict(gpu_memory_utilization=0.5, max_model_len=4096,+ max_num_seqs=8, enforce_eager=False)+ unknown = set(options) - defaults.keys()+ if unknown:+ raise ValueError(f"unknown vllm_options: {sorted(unknown)}")+ defaults.update(options)+ self.max_model_len = defaults["max_model_len"]+ self.token_map = owner.token_map+ lm_dir = owner.repo_dir / owner.config.get("lm_dir", "lm")+ config = AutoConfig.from_pretrained(str(lm_dir))+ if config.model_type != "qwen3":+ raise ValueError("the vLLM backend currently supports Qwen3 KittenTTS 2 checkpoints")+ owner.dtype = torch.bfloat16 if owner.config.get("dtype", "bf16") == "bf16" else torch.float32+ owner.tokenizer = AutoTokenizer.from_pretrained(str(lm_dir))+ state = owner._load_lm_weights(lm_dir)+ self.embedding = state["model.embed_tokens.weight"].to(owner.dtype)+ self.spk_proj = load_speaker_head(config.hidden_size, str(lm_dir),+ owner.device, owner.dtype,+ spk_dim=owner.config.get("speaker_embedding_dim", 512),+ state=state)+ del state+ with _engine_environment():+ from vllm import LLM+ from vllm.config import LoggingConfig++ self.engine = LLM(+ model=str(lm_dir), tokenizer=str(lm_dir), dtype=owner.dtype,+ load_format="kittenml",+ model_loader_extra_config={"repo_dir": str(owner.repo_dir),+ "config": owner.config, "weights": owner.weights},+ enable_prompt_embeds=True, enable_prefix_caching=False,+ logits_processors=["kittenml.kittentts2.vllm.logits:KittenLogitsProcessor"],+ logging_config=LoggingConfig(), disable_log_stats=True, **defaults)++ def prepare_request(self, prompt, speaker_embedding, settings, max_new_tokens, advanced):+ from vllm import SamplingParams++ if len(prompt) + int(max_new_tokens) > self.max_model_len:+ raise ValueError("prompt and token budget exceed vllm_options['max_model_len']; "+ "increase it or shorten the reference/token budget")+ with torch.inference_mode():+ embeds = self.embedding[torch.tensor(prompt, dtype=torch.long)].clone()+ embeds[0] = self.spk_proj(speaker_embedding).squeeze(0).cpu()+ sampling = {k: settings[k] for k in ("temperature", "top_k", "top_p")}+ sampling["min_p"] = settings.get("min_p", 0.0)+ sampling.update({k: advanced[k] for k in ("repetition_penalty", "repetition_window",+ "token_run_penalty", "token_run_grace")})+ seed = advanced.get("seed")+ if seed is None:+ seed = int(torch.randint(0, 2**31 - 1, ()).item())+ params = SamplingParams(+ temperature=1.0, top_k=-1, top_p=1.0, min_p=0.0,+ repetition_penalty=1.0, max_tokens=int(max_new_tokens),+ stop_token_ids=[self.token_map.speech_end_id, self.token_map.stop_id],+ ignore_eos=True, detokenize=False, seed=int(seed),+ extra_args={"kittenml": {"prompt": prompt, "token_map": asdict(self.token_map),+ "sampling": sampling}})+ return {"prompt_embeds": embeds}, params++ def generate_tokens(self, prompt, speaker_embedding, settings, max_new_tokens, advanced):+ request, params = self.prepare_request(prompt, speaker_embedding, settings,+ max_new_tokens, advanced)+ output = self.engine.generate([request], [params], use_tqdm=False)[0]+ low, high = self.token_map.audio_id_base, self.token_map.audio_id_end+ return [t - low for t in output.outputs[0].token_ids if low <= t < high]diff --git a/kittenml/kittentts2/vllm/loader.py b/kittenml/kittentts2/vllm/loader.pynew file mode 100644index 0000000..96b01f7--- /dev/null+++ b/kittenml/kittentts2/vllm/loader.py@@ -0,0 +1,25 @@+"""vLLM plugin: selected KittenTTS weights, without an intermediate export."""++from vllm.model_executor.model_loader.base_loader import BaseModelLoader++from ..weights import load_lm_weights+++class KittenModelLoader(BaseModelLoader):+ def download_model(self, model_config):+ # KittenTTS resolves the selected files before constructing the engine.+ pass++ def load_weights(self, model, model_config):+ options = self.load_config.model_loader_extra_config+ state = load_lm_weights(options["repo_dir"], options["config"], options["weights"])+ loaded = model.load_weights((name, tensor) for name, tensor in state.items()+ if not name.startswith("spk_proj."))+ missing = set(dict(model.named_parameters())) - loaded+ if missing:+ raise ValueError(f"weights missing from the checkpoint: {sorted(missing)[:5]}")+++def register():+ from vllm.model_executor.model_loader import register_model_loader+ register_model_loader("kittenml")(KittenModelLoader)diff --git a/kittenml/kittentts2/vllm/logits.py b/kittenml/kittentts2/vllm/logits.pynew file mode 100644index 0000000..bcc0f4a--- /dev/null+++ b/kittenml/kittentts2/vllm/logits.py@@ -0,0 +1,40 @@+"""KittenTTS's complete sampling pipeline for vLLM Model Runner V2."""++import torch+from vllm.v1.worker.gpu.sample.logits_processor import LogitsProcessor++from ..logits import build_logits_processors+from ..tokens import TokenMap+++class KittenLogitsProcessor(LogitsProcessor):+ def __init__(self, vllm_config, req_states):+ self.req_states = req_states+ self.requests = {}++ def add_request(self, req_idx, sampling_params):+ extra = (sampling_params.extra_args or {}).get("kittenml")+ self.requests.pop(req_idx, None)+ if extra is None:+ return False+ prompt = torch.tensor(extra["prompt"], device=self.req_states.device,+ dtype=torch.long)+ processors = build_logits_processors(+ TokenMap(**extra["token_map"]), **extra["sampling"],+ prompt_length=len(extra["prompt"]))+ self.requests[req_idx] = (prompt, processors)+ return True++ def apply(self, logits, ctx):+ # No speculative decoding: each row corresponds to one active request,+ # and the host sequence lengths are exact (no per-token GPU readback).+ for row, req_idx in enumerate(ctx.idx_mapping_np):+ request = self.requests.get(int(req_idx))+ if request is None:+ continue+ prompt, processors = request+ length = int(ctx.seq_lens_upper_bound_np[row])+ generated = self.req_states.all_token_ids.gpu[int(req_idx), len(prompt):length]+ history = torch.cat((prompt, generated)).unsqueeze(0)+ logits[row:row + 1] = processors(history, logits[row:row + 1])+ return logitsdiff --git a/kittenml/kittentts2/weights.py b/kittenml/kittentts2/weights.pynew file mode 100644index 0000000..cc52c39--- /dev/null+++ b/kittenml/kittentts2/weights.py@@ -0,0 +1,48 @@+"""Read the selected checkpoint identically for Torch and vLLM."""++from pathlib import Path+++def available_weights(config):+ return ["full", *(name for name, key in (("packed", "lm_packed"), ("emb4", "lm_emb4"))+ if config.get(key))]+++def weights_path(repo_dir, config, name):+ if name == "full":+ return None+ key = {"packed": "lm_packed", "emb4": "lm_emb4"}.get(name)+ if key is None:+ raise ValueError(f"Unknown weights {name!r}. Choose from: {available_weights(config)}")+ declared = config.get(key)+ if not declared:+ raise ValueError(f"this repository declares no {name!r} weights")+ return Path(repo_dir) / declared+++def load_lm_weights(repo_dir, config, variant, lm_dir=None):+ """Unpack on CPU without re-quantizing or writing an expanded checkpoint."""+ from safetensors import safe_open+ from safetensors.torch import load_file++ path = weights_path(repo_dir, config, variant)+ if path is not None:+ if not path.is_file():+ raise FileNotFoundError(f"{variant!r} weights not found: {path}")+ if variant == "emb4":+ from .tl2_emb4 import load_state_dict+ return load_state_dict(path)+ with safe_open(str(path), framework="pt") as handle:+ metadata = handle.metadata() or {}+ tensors = {k: handle.get_tensor(k) for k in handle.keys()}+ from .ternary import unpack_state_dict+ return unpack_state_dict(tensors, metadata)++ lm_dir = Path(lm_dir) if lm_dir is not None else Path(repo_dir) / config.get("lm_dir", "lm")+ state = {}+ for shard in sorted(lm_dir.glob("model*.safetensors")):+ if "-" not in shard.stem:+ state.update(load_file(str(shard)))+ if not state:+ raise FileNotFoundError(f"no plain model weights in {lm_dir}")+ return statediff --git a/kittenml/kittentts_legacy/model.py b/kittenml/kittentts_legacy/model.pyindex e3c6020..d92510a 100644--- a/kittenml/kittentts_legacy/model.py+++ b/kittenml/kittentts_legacy/model.py@@ -25,7 +25,7 @@ class KittenTTSOnnx:"""def __init__(self, model_name="KittenML/kitten-tts-nano-0.8", cache_dir=None,- backend=None, device=None, hf_token=None):+ backend=None, device=None, hf_token=None, *, vllm_options=None):"""Initialize an ONNX KittenTTS model.Args:@@ -35,6 +35,8 @@ class KittenTTSOnnx:device: Unused here; accepted so both backends share a signaturehf_token: Unused here; accepted so both backends share a signature"""+ if vllm_options is not None:+ raise ValueError("the vLLM backend requires a KittenTTS 2 model")local_dir, repo_id, config = getattr(self, "_resolved", None) or \resolve_repo(model_name, cache_dir)self.model = build_onnx_model(config, local_dir, repo_id, cache_dir, backend)diff --git a/pyproject.toml b/pyproject.tomlindex a1323f4..d28404e 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"[project]name = "kittenml"-version = "0.9.3"+version = "0.9.4"description = "Text-to-speech with voice cloning and expression control, plus ultra-lightweight models that run on CPU"readme = "README.md"requires-python = ">=3.10"@@ -52,6 +52,12 @@ Homepage = "https://github.com/KittenML/KittenTTS"Repository = "https://github.com/KittenML/KittenTTS"Issues = "https://github.com/KittenML/KittenTTS/issues"+[project.optional-dependencies]+vllm = ["vllm==0.31.0; sys_platform == 'linux'"]++[project.entry-points."vllm.general_plugins"]+kittenml = "kittenml.kittentts2.vllm.loader:register"+[tool.setuptools.packages.find]where = ["."]include = ["kittenml*"]diff --git a/setup.py b/setup.pyindex 54b0d5d..34a57b8 100644--- a/setup.py+++ b/setup.py@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:setup(name="kittenml",- version="0.9.3",+ version="0.9.4",author="KittenML",author_email="",description="Text-to-speech with voice cloning and expression control, plus ultra-lightweight models that run on CPU",diff --git a/tests/test_vllm.py b/tests/test_vllm.pynew file mode 100644index 0000000..2a16ebd--- /dev/null+++ b/tests/test_vllm.py@@ -0,0 +1,294 @@+"""Small checkpoints and sampler parity; no GPU or model download required."""++from dataclasses import asdict+from types import SimpleNamespace+from unittest import mock++import numpy as np+import pytest+import torch+from safetensors.torch import save_file++from kittenml import KittenTTS+from kittenml.kittentts2.ternary import pack_state_dict+from kittenml.kittentts2.tokens import TokenMap+from kittenml.kittentts2.weights import load_lm_weights+++def test_default_backend_keeps_native_loading(tmp_path):+ from kittenml.kittentts2.model import KittenTTS2+ config = {"token_map": asdict(TokenMap(5, 10, 1, 2, 3, 4, 0, 15))}+ with mock.patch.object(KittenTTS2, "_load_language_model") as load, \+ mock.patch("kittenml.kittentts2.model.S3Codec"), \+ mock.patch("kittenml.kittentts2.model.SpeakerEncoder"), \+ mock.patch("kittenml.kittentts2.model.ReferenceTranscriber"):+ model = KittenTTS2(tmp_path, config, device="cpu")+ load.assert_called_once_with()+ assert model.backend == "torch" and model.device == "cpu"+++def test_selected_packed_weights_are_exact_and_full_ignores_packed_files(tmp_path):+ lm = tmp_path / "lm"+ lm.mkdir()+ state = {"model.layer.weight": torch.arange(256).remainder(3).reshape(2, 128)+ .sub(1).to(torch.bfloat16) * 0.125,+ "spk_proj.0.bias": torch.tensor([0.7, -0.2], dtype=torch.bfloat16)}+ save_file(state, lm / "model.safetensors")+ tensors, metadata = pack_state_dict(state)+ save_file(tensors, lm / "model-ternary.safetensors", metadata=metadata)+ config = {"lm_packed": "lm/model-ternary.safetensors"}+ for variant in ("packed", "full"):+ result = load_lm_weights(tmp_path, config, variant)+ assert result.keys() == state.keys()+ assert all(torch.equal(result[name], tensor) for name, tensor in state.items())+ with pytest.raises(ValueError, match="Unknown weights"):+ load_lm_weights(tmp_path, config, "invalid")+ with pytest.raises(ValueError, match="no 'emb4'"):+ load_lm_weights(tmp_path, config, "emb4")+++def test_native_full_loader_preserves_explicit_checkpoint_directory(tmp_path):+ from kittenml.kittentts2.model import KittenTTS2++ lm_dir = tmp_path / "custom-lm"+ lm_dir.mkdir()+ expected = {"model.weight": torch.tensor([0.25, -0.5], dtype=torch.bfloat16)}+ save_file(expected, lm_dir / "model.safetensors")+ owner = SimpleNamespace(repo_dir=tmp_path, config={}, weights="full")+ loaded = KittenTTS2._load_lm_weights(owner, lm_dir)+ assert loaded.keys() == expected.keys()+ torch.testing.assert_close(loaded["model.weight"], expected["model.weight"], rtol=0, atol=0)+++def test_shared_speaker_head_matches_native_prefill_and_leaves_decode_unchanged():+ from kittenml.kittentts2.speaker import attach_speaker_head, load_speaker_head++ class TinyLM(torch.nn.Module):+ def __init__(self):+ super().__init__()+ self.config = SimpleNamespace(hidden_size=4)+ self.embedding = torch.nn.Embedding(8, 4)++ def get_input_embeddings(self):+ return self.embedding++ def forward(self, input_ids=None, inputs_embeds=None, cache_position=None):+ return inputs_embeds++ projection = torch.nn.Sequential(torch.nn.Linear(2, 4), torch.nn.LayerNorm(4))+ state = {"spk_proj." + k: v for k, v in projection.state_dict().items()}+ model = attach_speaker_head(TinyLM(), "unused", "cpu", torch.float32,+ spk_dim=2, state=state)+ head = load_speaker_head(4, "unused", "cpu", torch.float32, spk_dim=2, state=state)+ speaker = torch.tensor([[0.5, -0.3]])+ model._pending_speaker_embedding = speaker+ ids = torch.tensor([[0, 3, 1]])+ original = model.embedding(ids)+ prefill = model(input_ids=ids, cache_position=torch.tensor([0]))+ torch.testing.assert_close(prefill[:, 0], head(speaker))+ torch.testing.assert_close(prefill[:, 1:], original[:, 1:])+ decode = model(input_ids=ids, cache_position=torch.tensor([3]))+ torch.testing.assert_close(decode, original)+ with pytest.raises(ValueError, match="no spk_proj"):+ load_speaker_head(4, "unused", "cpu", torch.float32, spk_dim=2, state={})+++def test_emb4_unpacking_is_independent_of_vllm_default_dtype():+ from kittenml.kittentts2.tl2_emb4 import dequant_embedding+ tensors = {+ "emb4/tier_map": torch.zeros(2, dtype=torch.uint8),+ "emb4/t0_codes": torch.zeros(1, dtype=torch.uint8),+ "emb4/t0_centers": torch.tensor([[1., 2.], [3., 4.]], dtype=torch.float32),+ "emb4/t0_s": torch.ones(2, dtype=torch.float32),+ "emb4/t0_t": torch.ones(2, dtype=torch.float32),+ }+ metadata = {"V": "2", "D": "2", "Ks": "[2]"}+ previous = torch.get_default_dtype()+ try:+ torch.set_default_dtype(torch.float32)+ expected = dequant_embedding(tensors.__getitem__, metadata)+ torch.set_default_dtype(torch.bfloat16)+ actual = dequant_embedding(tensors.__getitem__, metadata)+ finally:+ torch.set_default_dtype(previous)+ assert actual.dtype == torch.bfloat16 and torch.equal(actual, expected)+++def test_optional_backend_errors_before_model_download():+ resolved = (None, "example/repo", {"type": "KITTEN2"})+ with mock.patch("kittenml.get_model.resolve_repo", return_value=resolved), \+ mock.patch("kittenml.kittentts2.vllm.backend.importlib.util.find_spec", return_value=None), \+ mock.patch("kittenml.get_model.snapshot_download") as download:+ with pytest.raises(ImportError, match=r"kittenml\[vllm\]"):+ KittenTTS("example/repo", backend="vllm")+ download.assert_not_called()+ with mock.patch("kittenml.get_model.resolve_repo", return_value=(None, "legacy", {"type": "ONNX1"})):+ with pytest.raises(ValueError, match="KittenTTS 2"):+ KittenTTS("legacy", backend="vllm")+++def test_vllm_options_require_backend():+ with mock.patch("kittenml.get_model.resolve_repo", return_value=(None, "model", {"type": "KITTEN2"})):+ with pytest.raises(ValueError, match="requires backend"):+ KittenTTS("model", vllm_options={})+++def test_explicit_default_options_do_not_break_legacy_models():+ resolved = (None, "legacy", {"type": "ONNX1"})+ with mock.patch("kittenml.get_model.resolve_repo", return_value=resolved), \+ mock.patch("kittenml.kittentts_legacy.model.build_onnx_model") as build:+ model = KittenTTS("legacy", vllm_options=None)+ assert model.model is build.return_value+ build.assert_called_once_with(resolved[2], None, "legacy", None, None)+++def test_environment_rejects_unsupported_devices_before_engine_start():+ from kittenml.kittentts2.vllm.backend import validate_environment+ with mock.patch("kittenml.kittentts2.vllm.backend.sys.platform", "win32"):+ with pytest.raises(RuntimeError, match="Linux"):+ validate_environment()+ with mock.patch("kittenml.kittentts2.vllm.backend.importlib.util.find_spec", return_value=object()), \+ mock.patch("torch.cuda.is_available", return_value=False):+ with pytest.raises(RuntimeError, match="CUDA GPU"):+ validate_environment()+ with mock.patch("kittenml.kittentts2.vllm.backend.importlib.util.find_spec", return_value=object()), \+ mock.patch("torch.cuda.is_available", return_value=True):+ with pytest.raises(RuntimeError, match="CUDA GPU"):+ validate_environment("cpu")+ with pytest.raises(ValueError, match="CUDA_VISIBLE_DEVICES"):+ validate_environment("cuda:1")++[email protected]("key, default, override", [+ ("VLLM_USE_FLASHINFER_SAMPLER", "0", "1"),+ ("VLLM_LOGGING_LEVEL", "WARNING", "INFO"),+ ("TQDM_DISABLE", "1", "0"),+])+def test_engine_environment_preserves_user_choice_and_restores_after_failure(+ monkeypatch, key, default, override):+ import os+ from kittenml.kittentts2.vllm.backend import _engine_environment+ monkeypatch.delenv(key, raising=False)+ with pytest.raises(RuntimeError):+ with _engine_environment():+ assert os.environ[key] == default+ raise RuntimeError("engine failed")+ assert key not in os.environ+ monkeypatch.setenv(key, override)+ with _engine_environment():+ assert os.environ[key] == override+ assert os.environ[key] == override+++def test_engine_receives_quiet_logging_config_without_changing_native_options(monkeypatch, tmp_path):+ pytest.importorskip("vllm")+ import os+ from kittenml.kittentts2.vllm.backend import VLLMBackend++ monkeypatch.delenv("VLLM_LOGGING_LEVEL", raising=False)+ owner = SimpleNamespace(device="cuda", repo_dir=tmp_path, config={}, weights="packed",+ token_map=None, _load_lm_weights=mock.Mock(return_value={+ "model.embed_tokens.weight": torch.zeros(2, 4)}))++ def create_engine(**kwargs):+ assert kwargs["logging_config"].log_level == "WARNING"+ assert os.environ["VLLM_LOGGING_LEVEL"] == "WARNING"+ assert os.environ["TQDM_DISABLE"] == "1"+ assert kwargs["enable_prompt_embeds"] and not kwargs["enable_prefix_caching"]+ assert kwargs["model_loader_extra_config"]["weights"] == "packed"+ return mock.sentinel.engine++ with mock.patch("kittenml.kittentts2.vllm.backend.validate_environment"), \+ mock.patch("transformers.AutoConfig.from_pretrained", return_value=SimpleNamespace(+ model_type="qwen3", hidden_size=4)), \+ mock.patch("transformers.AutoTokenizer.from_pretrained"), \+ mock.patch("kittenml.kittentts2.speaker.load_speaker_head"), \+ mock.patch("vllm.LLM", side_effect=create_engine):+ backend = VLLMBackend(owner)+ assert backend.engine is mock.sentinel.engine+ assert "VLLM_LOGGING_LEVEL" not in os.environ+++def test_vllm_loader_filters_speaker_weights_and_rejects_missing_parameters():+ pytest.importorskip("vllm")+ from kittenml.kittentts2.vllm.loader import KittenModelLoader+ loader = object.__new__(KittenModelLoader)+ loader.load_config = SimpleNamespace(model_loader_extra_config={+ "repo_dir": "unused", "config": {}, "weights": "packed"})++ class TinyModel:+ def load_weights(self, weights):+ self.loaded = dict(weights)+ return set(self.loaded)++ def named_parameters(self):+ return [("model.weight", None)]++ model = TinyModel()+ state = {"model.weight": torch.ones(2), "spk_proj.0.bias": torch.zeros(2)}+ with mock.patch("kittenml.kittentts2.vllm.loader.load_lm_weights", return_value=state):+ loader.load_weights(model, None)+ assert set(model.loaded) == {"model.weight"}+ with mock.patch("kittenml.kittentts2.vllm.loader.load_lm_weights", return_value={}):+ with pytest.raises(ValueError, match="missing from the checkpoint"):+ loader.load_weights(model, None)++[email protected]("window", [None, 3])+def test_sampling_matches_native_when_requests_reorder_and_slots_are_reused(window):+ pytest.importorskip("vllm")+ from kittenml.kittentts2.logits import build_logits_processors+ from kittenml.kittentts2.vllm.logits import KittenLogitsProcessor++ tm = TokenMap(audio_id_base=5, num_audio_tokens=5000, speech_start_id=1,+ speech_end_id=2, text_start_id=3, text_end_id=4, start_id=0, stop_id=5005)+ sampling = dict(temperature=0.8, top_k=60, top_p=0.9, min_p=0.05,+ repetition_penalty=1.1, repetition_window=window,+ token_run_penalty=1.3, token_run_grace=3)+ original = {0: [0, 3, 7, 1], 1: [0, 3, 8, 9, 1]}+ history = {0: original[0] + [7, 7, 7, 7],+ 1: original[1] + [tm.audio_id_base + 4299] * 4}+ buf = torch.zeros(2, 16, dtype=torch.long)+ # Embedding prompts may have placeholder IDs inside vLLM. The processor must+ # restore the real prompt before applying full-history repetition penalties.+ for slot in history:+ buf[slot, len(original[slot]):len(history[slot])] = torch.tensor(history[slot][len(original[slot]):])+ states = SimpleNamespace(device=torch.device("cpu"), all_token_ids=SimpleNamespace(gpu=buf))+ processor = KittenLogitsProcessor(None, states)+ for slot in original:+ params = SimpleNamespace(extra_args={"kittenml": {"prompt": original[slot],+ "token_map": asdict(tm), "sampling": sampling}})+ assert processor.add_request(slot, params)+ ctx = SimpleNamespace(idx_mapping_np=np.array([1, 0]),+ seq_lens_upper_bound_np=np.array([len(history[1]), len(history[0])]))+ scores = torch.randn(2, 5006, generator=torch.Generator().manual_seed(7))+ expected = scores.clone()+ for row, slot in enumerate([1, 0]):+ native = build_logits_processors(tm, **sampling, prompt_length=len(original[slot]))+ expected[row:row + 1] = native(torch.tensor([history[slot]]), expected[row:row + 1])+ torch.testing.assert_close(processor.apply(scores, ctx), expected)+ assert not processor.add_request(0, SimpleNamespace(extra_args=None))+ assert 0 not in processor.requests+++def test_sampler_is_neutral_and_speaker_replaces_only_first_embedding():+ pytest.importorskip("vllm")+ from kittenml.kittentts2.vllm.backend import VLLMBackend++ backend = object.__new__(VLLMBackend)+ backend.max_model_len = 32+ backend.token_map = TokenMap(5, 10, 1, 2, 3, 4, 0, 15)+ backend.embedding = torch.arange(64, dtype=torch.float32).reshape(16, 4)+ backend.spk_proj = torch.nn.Identity()+ settings = dict(temperature=0.7, top_k=5, top_p=0.8, min_p=0.02)+ advanced = dict(seed=42, repetition_penalty=1.2, repetition_window=3,+ token_run_penalty=1.5, token_run_grace=4)+ request, params = backend.prepare_request([0, 3, 1], torch.ones(1, 4), settings, 8, advanced)+ torch.testing.assert_close(request["prompt_embeds"][0], torch.ones(4))+ torch.testing.assert_close(request["prompt_embeds"][1:], backend.embedding[[3, 1]])+ assert params.temperature == params.top_p == params.repetition_penalty == 1.0+ assert params.top_k == -1 and params.min_p == 0.0 and params.seed == 42+ assert params.stop_token_ids == [2, 15] and params.ignore_eos+ assert params.extra_args["kittenml"]["sampling"]["temperature"] == 0.7+ with pytest.raises(ValueError, match="token budget"):+ backend.prepare_request([0, 3, 1], torch.ones(1, 4), settings, 30, advanced)