Skip to content

Open-source State-of-the-art TTS model which runs on a CPU 😻

Python

Add optional vLLM acceleration for KittenTTS 2

haaziq-sl471f8430724b
diff --git a/README.md b/README.md
index 27e377b..6453233 100644
--- a/README.md
+++ b/README.md
@@ -32,6 +32,7 @@ The library also ships the original [**lightweight legacy models**](docs/onnx-mo
- [Voice cloning](#voice-cloning)
- [Expression controls](#expression-controls)
- [Running on CPU](#running-on-cpu)
+- [Running with vLLM](#running-with-vllm)
- [Long text and streaming](#long-text-and-streaming)
- [Documentation](#documentation)
- [System Requirements](#system-requirements)
@@ -185,6 +186,32 @@ KittenTTS 2 runs on CPU out of the box β€” `device` is auto-detected β€” but the
[kitten-tts-2-cpp](https://github.com/KittenML/kitten-tts-2-cpp), our llama.cpp fork. It reads
the GGUF weights in the model repository's `cpp/` directory.
+## Running with vLLM
+
+For GPU inference with vLLM on Linux and an NVIDIA CUDA GPU, install the optional backend:
+
+```bash
+pip install "kittenml[vllm]"
+```
+
+Select `backend="vllm"`:
+
+```python
+from kittenml import KittenTTS
+
+def main():
+ m = KittenTTS("KittenML/kitten-tts-2", backend="vllm")
+ m.generate_to_file("Hello there.", "output.wav", voice="Bruno")
+ print("Saved output.wav")
+
+if __name__ == "__main__":
+ main()
+```
+
+Keep the main guard when running a script: vLLM starts a worker process. The same
+voices, decoders, weight choices and generation arguments apply. See the
+[vLLM guide](docs/vllm.md) for memory settings and benchmarks.
+
## Long text and streaming
diff --git a/docs/README.md b/docs/README.md
index a575890..33d92ff 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -7,6 +7,7 @@ Fuller detail than the [project README](../README.md), which covers getting star
| [api.md](api.md) | The KittenTTS 2 API: every argument to `generate`, streaming, and the advanced knobs |
| [voices-and-expression.md](voices-and-expression.md) | The 47 built-in voices, the emotion and vocal-event tags, the ten languages |
| [decoders.md](decoders.md) | How audio is decoded, and the smaller quantised decoders |
+| [vllm.md](vllm.md) | Optional NVIDIA GPU inference with vLLM |
| [text-normalization.md](text-normalization.md) | How written text becomes spoken text |
| [architecture.md](architecture.md) | What the model is, package layout, vendored components |
| [onnx-models.md](onnx-models.md) | The lightweight CPU models and their API |
diff --git a/docs/api.md b/docs/api.md
index a02dd5c..63404f9 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -14,6 +14,8 @@ Loads whichever model family the repository's `config.json` declares.
| `hf_token` | `str` | `None` | Token, only for downloading a private model repository |
| `decoder` | `str` | repo default | Which decoder to load; see [decoders.md](decoders.md) |
| `weights` | `str` | `"packed"` | Which weight packing to load: `"packed"` (lossless), `"emb4"` (smaller, quantised embedding), `"full"` (bf16) |
+| `backend` | `str` | `None` | Default PyTorch inference, or `"vllm"` for optional NVIDIA GPU inference on Linux |
+| `vllm_options` | `dict` | `None` | Engine memory and context settings; see [vLLM](vllm.md) |
# `model.generate(text, voice=None, reference=None, ...)`
diff --git a/docs/vllm.md b/docs/vllm.md
new file mode 100644
index 0000000..b0d32c6
--- /dev/null
+++ b/docs/vllm.md
@@ -0,0 +1,72 @@
+# Running with vLLM
+
+The optional `vllm` backend runs KittenTTS 2's language model on an NVIDIA CUDA
+GPU. Audio decoding, voice cloning and text preparation use the existing
+KittenTTS implementations. The default backend remains PyTorch and already
+supports CUDA.
+
+Install the optional backend and select `backend="vllm"` as shown in the
+[README](../README.md#running-with-vllm). This backend requires Linux and uses
+vLLM 0.31.0; see its [GPU requirements](https://docs.vllm.ai/en/v0.31.0/getting_started/installation/gpu/)
+for compatible NVIDIA drivers.
+It loads the selected published weights directly; no manual export
+or additional language-model checkpoint is required. Packed weights are expanded in
+memory, so their smaller download size does not reduce GPU memory use.
+
+The engine defaults to a 4,096-token context, up to eight active sequences, and
+half the GPU memory, leaving space for audio decoding. In the README example,
+pass `vllm_options` when constructing the model to change these settings:
+
+```python
+m = KittenTTS("KittenML/kitten-tts-2", backend="vllm",
+ vllm_options={"gpu_memory_utilization": 0.5,
+ "max_model_len": 4096})
+```
+
+`gpu_memory_utilization` is a fraction of the GPU's total VRAM. It sets vLLM's
+budget; the audio decoder and other GPU processes need additional memory.
+
+If VRAM is limited, reduce `max_model_len` to lower cache requirements. If startup
+reports no available memory for cache blocks, increase `gpu_memory_utilization`
+to give vLLM a larger share of VRAM, leaving space for audio decoding and other
+GPU processes.
+
+Routine engine logs are quiet by default; warnings and errors remain visible.
+Set `VLLM_LOGGING_LEVEL=INFO` before starting Python to see startup diagnostics.
+
+The context must fit the reference prompt and the requested generation budget.
+Long text still uses KittenTTS's sentence chunking. `generate_stream` yields
+completed audio chunks, just like the default backend.
+
+`vllm_options` also accepts `max_num_seqs` and `enforce_eager`. Select a GPU with
+`CUDA_VISIBLE_DEVICES` before starting Python. Sampling implementations differ
+between the backends, so a matching seed does not guarantee identical audio.
+
+## Benchmarks
+
+Measured on an RTX 3060 (12 GB), Linux, Python 3.12 and NVIDIA driver 580.95.05,
+using `weights="packed"`, the default decoder and the stable preset. These are
+median end-to-end synthesis times from five runs after two warmups per voice.
+Short, medium and long inputs contain 61, 226 and 680 characters; the long input
+is split into two chunks. Model initialization and file writes are excluded.
+
+| Input / voice | PyTorch GPU | vLLM | Speedup |
+|---|---:|---:|---:|
+| Short / Bruno | 6.01 s | 2.05 s | 2.93x |
+| Short / Bella | 6.98 s | 2.54 s | 2.75x |
+| Medium / Bruno | 22.93 s | 5.49 s | 4.18x |
+| Medium / Bella | 27.95 s | 8.26 s | 3.39x |
+| Long / Bruno | 56.77 s | 17.56 s | 3.23x |
+| Long / Bella | 79.24 s | 24.04 s | 3.30x |
+
+The PyTorch column uses unchanged upstream code in a fresh normal installation
+(PyTorch 2.14.1, Transformers 5.18.0). With versions matched to the vLLM install
+(PyTorch 2.13.0, Transformers 5.17.0, vLLM 0.31.0), the measured speedup was
+3.05-4.49x. Generated audio lengths differ between samplers; vLLM's synthesis
+time was 0.44-0.48 seconds per second of output audio in these tests.
+
+vLLM used about 8.5 GiB of GPU memory versus 5.8 GiB for the normal installation.
+Cloning without `reference_text` also loads Whisper; those tests used up to
+11.5 GiB. Supplying the transcript skips that model.
+Initialization with cached downloads took 51 seconds versus 10 seconds; the
+first vLLM startup took about 162 seconds. Reuse the loaded model across calls.
diff --git a/kittenml/get_model.py b/kittenml/get_model.py
index 54c6188..b76afa1 100644
--- a/kittenml/get_model.py
+++ b/kittenml/get_model.py
@@ -63,25 +63,35 @@ class KittenTTS(KittenTTSOnnx):
"""
def __new__(cls, model_name="KittenML/kitten-tts-nano-0.8", cache_dir=None,
- backend=None, device=None, hf_token=None, decoder=None, weights=None):
+ backend=None, device=None, hf_token=None, decoder=None, weights=None,
+ vllm_options=None):
local_dir, repo_id, config = resolve_repo(model_name, cache_dir)
model_type = config.get("type")
if model_type == KITTEN2_TYPE:
from .kittentts2 import KittenTTS2
+ if backend == "vllm":
+ from .kittentts2.vllm.backend import validate_environment
+ validate_environment(device)
+ elif vllm_options is not None:
+ raise ValueError("vllm_options requires backend='vllm'")
if local_dir is None:
local_dir = snapshot_download(
repo_id=repo_id, cache_dir=cache_dir, token=hf_token,
allow_patterns=_kitten2_files(config, decoder, weights))
# Returning a different class means __init__ below is never called.
return KittenTTS2(local_dir, config, device=device, cache_dir=cache_dir,
- hf_token=hf_token, decoder=decoder, weights=weights)
+ hf_token=hf_token, decoder=decoder, weights=weights,
+ backend=backend, vllm_options=vllm_options)
if model_type not in ONNX_TYPES:
raise ValueError(
f"Unsupported model type {model_type!r}; expected one of "
f"{[*ONNX_TYPES, KITTEN2_TYPE]}")
+ if backend == "vllm" or vllm_options is not None:
+ raise ValueError("the vLLM backend requires a KittenTTS 2 model")
+
instance = super().__new__(cls)
# Hand the already-resolved repository to KittenTTSOnnx.__init__ rather
# than making it hit the Hub a second time.
diff --git a/kittenml/kittentts2/model.py b/kittenml/kittentts2/model.py
index 60db654..8d356ee 100644
--- a/kittenml/kittentts2/model.py
+++ b/kittenml/kittentts2/model.py
@@ -92,7 +92,7 @@ class KittenTTS2:
"""
def __init__(self, repo_dir, config, device=None, cache_dir=None, hf_token=None,
- decoder=None, weights=None):
+ decoder=None, weights=None, backend=None, vllm_options=None):
self.repo_dir = pathlib.Path(repo_dir)
self.config = config
self.cache_dir = cache_dir
@@ -112,7 +112,12 @@ class KittenTTS2:
self.chunk_min_chars = generation.get("chunk_min_chars", text_utils.CHUNK_MIN_CHARS)
self.weights = weights or config.get("default_weights", "packed")
- self._load_language_model()
+ self.backend = "vllm" if backend == "vllm" else "torch"
+ if self.backend == "vllm":
+ from .vllm.backend import VLLMBackend
+ self.model = VLLMBackend(self, vllm_options)
+ else:
+ self._load_language_model()
self.decoders = config.get("decoders", {})
self.decoder = decoder or config.get("default_decoder", "default")
self.codec = S3Codec(device=self.device, cache_dir=cache_dir,
@@ -233,26 +238,15 @@ class KittenTTS2:
return model
def _weights_path(self, name):
- """Where a named weight variant lives, or None to read the plain files."""
- if name == "full":
- return None
- key = {"packed": "lm_packed", "emb4": "lm_emb4"}.get(name)
- if key is None:
- raise ValueError(f"Unknown weights {name!r}. "
- f"Choose from: {self.available_weights}")
- declared = self.config.get(key)
- if not declared:
- raise ValueError(f"this repository declares no {name!r} weights")
- return self.repo_dir / declared
+ """Where a named weight variant lives, or None for the plain files."""
+ from .weights import weights_path
+ return weights_path(self.repo_dir, self.config, name)
@property
def available_weights(self):
"""Weight variants this repository offers."""
- names = ["full"]
- for name, key in (("packed", "lm_packed"), ("emb4", "lm_emb4")):
- if self.config.get(key):
- names.append(name)
- return names
+ from .weights import available_weights
+ return available_weights(self.config)
def _load_lm_weights(self, lm_dir):
"""The language model's weights, in whichever packing was selected.
@@ -262,30 +256,8 @@ class KittenTTS2:
"emb4" is smaller again but quantises the embedding, which is lossy β€”
see tl2_emb4.py. "full" reads the plain bf16 files.
"""
- from safetensors import safe_open
- from safetensors.torch import load_file
-
- path = self._weights_path(self.weights)
- if path is not None:
- if not path.is_file():
- raise FileNotFoundError(f"{self.weights!r} weights not found: {path}")
- if self.weights == "emb4":
- from .tl2_emb4 import load_state_dict
- return load_state_dict(path)
- with safe_open(str(path), framework="pt") as handle:
- metadata = handle.metadata() or {}
- tensors = {k: handle.get_tensor(k) for k in handle.keys()}
- from .ternary import unpack_state_dict
- return unpack_state_dict(tensors, metadata)
-
- state = {}
- for shard in sorted(pathlib.Path(lm_dir).glob("model*.safetensors")):
- if "-" in shard.stem: # a packed variant, not a plain shard
- continue
- state.update(load_file(str(shard)))
- if not state:
- raise FileNotFoundError(f"no plain model weights in {lm_dir}")
- return state
+ from .weights import load_lm_weights
+ return load_lm_weights(self.repo_dir, self.config, self.weights, lm_dir=lm_dir)
def _load_voices(self):
voices_path = self.repo_dir / self.config.get("voices", "voices/voices.json")
@@ -385,6 +357,10 @@ class KittenTTS2:
reference_prefix=reference["prefix"] if use_reference_prompt else None,
emotion_ids=emotion_ids)
+ if self.backend == "vllm":
+ return self.model.generate_tokens(prompt, reference["embedding"], settings,
+ max_new_tokens, advanced)
+
processors = build_logits_processors(
self.token_map,
temperature=settings["temperature"], top_k=settings["top_k"],
diff --git a/kittenml/kittentts2/speaker.py b/kittenml/kittentts2/speaker.py
index 9abac2f..dff8756 100644
--- a/kittenml/kittentts2/speaker.py
+++ b/kittenml/kittentts2/speaker.py
@@ -24,19 +24,11 @@ from .speaker_embedding import SAMPLE_RATE
WHISPER_MODEL = "openai/whisper-large-v3"
-def attach_speaker_head(model, checkpoint_dir, device, dtype,
- spk_dim=SPEAKER_EMBEDDING_DIM, state=None):
- """Restore `model.spk_proj` from the checkpoint and wire up its prefill hook.
-
- Without this the model runs unconditioned β€” it still produces speech, but the
- voice is arbitrary, so a missing head is an error rather than a warning.
-
- `state` lets a caller pass weights it has already read, which avoids loading
- several gigabytes twice just to find four tensors.
- """
- hidden = int(model.config.hidden_size)
- model.spk_proj = nn.Sequential(nn.Linear(spk_dim, hidden, bias=True),
- nn.LayerNorm(hidden)).to(dtype=dtype)
+def load_speaker_head(hidden, checkpoint_dir, device, dtype,
+ spk_dim=SPEAKER_EMBEDDING_DIM, state=None):
+ """Restore the trained projection without constructing a language model."""
+ head = nn.Sequential(nn.Linear(spk_dim, int(hidden), bias=True),
+ nn.LayerNorm(int(hidden))).to(dtype=dtype)
if state is None:
state = {}
@@ -48,8 +40,19 @@ def attach_speaker_head(model, checkpoint_dir, device, dtype,
raise ValueError(
f"no spk_proj.* weights in {checkpoint_dir} β€” this checkpoint cannot be "
"speaker-conditioned, and generating from it would ignore the voice")
- model.spk_proj.load_state_dict(head_state, strict=True)
- model.spk_proj.to(device=device, dtype=dtype).eval()
+ head.load_state_dict(head_state, strict=True)
+ return head.to(device=device, dtype=dtype).eval()
+
+
+def attach_speaker_head(model, checkpoint_dir, device, dtype,
+ spk_dim=SPEAKER_EMBEDDING_DIM, state=None):
+ """Restore `model.spk_proj` and inject it at prefill.
+
+ A missing head is an error: speech would otherwise ignore the chosen voice.
+ Pass an already-loaded `state` to avoid reading the checkpoint twice.
+ """
+ model.spk_proj = load_speaker_head(model.config.hidden_size, checkpoint_dir,
+ device, dtype, spk_dim=spk_dim, state=state)
def _inject_speaker(module, args, kwargs):
embedding = kwargs.pop("speaker_embedding", None)
diff --git a/kittenml/kittentts2/tl2_emb4.py b/kittenml/kittentts2/tl2_emb4.py
index ccbb419..9c31aea 100644
--- a/kittenml/kittentts2/tl2_emb4.py
+++ b/kittenml/kittentts2/tl2_emb4.py
@@ -87,7 +87,7 @@ def unpack_tl2(buf, out_features, in_features):
def _hadamard(n):
- matrix = torch.tensor([[1.0]])
+ matrix = torch.tensor([[1.0]], dtype=torch.float32)
for _ in range(int(round(math.log2(n)))):
matrix = torch.cat([torch.cat([matrix, matrix], 1),
torch.cat([matrix, -matrix], 1)], 0)
diff --git a/kittenml/kittentts2/vllm/__init__.py b/kittenml/kittentts2/vllm/__init__.py
new file mode 100644
index 0000000..780a557
--- /dev/null
+++ b/kittenml/kittentts2/vllm/__init__.py
@@ -0,0 +1 @@
+"""Optional vLLM integration for KittenTTS 2."""
diff --git a/kittenml/kittentts2/vllm/backend.py b/kittenml/kittentts2/vllm/backend.py
new file mode 100644
index 0000000..25deebb
--- /dev/null
+++ b/kittenml/kittentts2/vllm/backend.py
@@ -0,0 +1,116 @@
+"""Optional GPU language-model backend; waveform decoding stays in KittenTTS."""
+
+from dataclasses import asdict
+import contextlib
+import importlib.util
+import os
+import sys
+
+import torch
+
+
+def _engine_environment():
+ # KittenTTS already applies its warpers. Use vLLM's native sampler by default
+ # so a pip installation needs no nvcc for FlashInfer's sampling JIT. The
+ # worker inherits these settings, including quiet startup defaults. Restore
+ # the caller's environment afterwards and respect explicit overrides.
+ defaults = {"VLLM_USE_FLASHINFER_SAMPLER": "0",
+ "VLLM_LOGGING_LEVEL": "WARNING", "TQDM_DISABLE": "1"}
+ previous = {key: os.environ.get(key) for key in defaults}
+ for key, value in defaults.items():
+ os.environ.setdefault(key, value)
+ try:
+ yield
+ finally:
+ for key, value in previous.items():
+ if value is None:
+ os.environ.pop(key, None)
+ else:
+ os.environ[key] = value
+
+
+def validate_environment(device=None):
+ if sys.platform != "linux":
+ raise RuntimeError("the vLLM backend requires Linux and an NVIDIA CUDA GPU")
+ if importlib.util.find_spec("vllm") is None:
+ raise ImportError('install the optional backend with pip install "kittenml[vllm]"')
+ selected = torch.device(device or "cuda")
+ if selected.type != "cuda" or not torch.cuda.is_available():
+ raise RuntimeError("the vLLM backend requires an NVIDIA CUDA GPU")
+ if selected.index not in (None, 0):
+ raise ValueError("select the GPU with CUDA_VISIBLE_DEVICES and use device='cuda'")
+
+
+class VLLMBackend:
+ def __init__(self, owner, options=None):
+ validate_environment(owner.device)
+ from transformers import AutoConfig, AutoTokenizer
+ from ..speaker import load_speaker_head
+
+ options = dict(options or {})
+ defaults = dict(gpu_memory_utilization=0.5, max_model_len=4096,
+ max_num_seqs=8, enforce_eager=False)
+ unknown = set(options) - defaults.keys()
+ if unknown:
+ raise ValueError(f"unknown vllm_options: {sorted(unknown)}")
+ defaults.update(options)
+ self.max_model_len = defaults["max_model_len"]
+ self.token_map = owner.token_map
+ lm_dir = owner.repo_dir / owner.config.get("lm_dir", "lm")
+ config = AutoConfig.from_pretrained(str(lm_dir))
+ if config.model_type != "qwen3":
+ raise ValueError("the vLLM backend currently supports Qwen3 KittenTTS 2 checkpoints")
+ owner.dtype = torch.bfloat16 if owner.config.get("dtype", "bf16") == "bf16" else torch.float32
+ owner.tokenizer = AutoTokenizer.from_pretrained(str(lm_dir))
+ state = owner._load_lm_weights(lm_dir)
+ self.embedding = state["model.embed_tokens.weight"].to(owner.dtype)
+ self.spk_proj = load_speaker_head(config.hidden_size, str(lm_dir),
+ owner.device, owner.dtype,
+ spk_dim=owner.config.get("speaker_embedding_dim", 512),
+ state=state)
+ del state
+ with _engine_environment():
+ from vllm import LLM
+ from vllm.config import LoggingConfig
+
+ self.engine = LLM(
+ model=str(lm_dir), tokenizer=str(lm_dir), dtype=owner.dtype,
+ load_format="kittenml",
+ model_loader_extra_config={"repo_dir": str(owner.repo_dir),
+ "config": owner.config, "weights": owner.weights},
+ enable_prompt_embeds=True, enable_prefix_caching=False,
+ logits_processors=["kittenml.kittentts2.vllm.logits:KittenLogitsProcessor"],
+ logging_config=LoggingConfig(), disable_log_stats=True, **defaults)
+
+ def prepare_request(self, prompt, speaker_embedding, settings, max_new_tokens, advanced):
+ from vllm import SamplingParams
+
+ if len(prompt) + int(max_new_tokens) > self.max_model_len:
+ raise ValueError("prompt and token budget exceed vllm_options['max_model_len']; "
+ "increase it or shorten the reference/token budget")
+ with torch.inference_mode():
+ embeds = self.embedding[torch.tensor(prompt, dtype=torch.long)].clone()
+ embeds[0] = self.spk_proj(speaker_embedding).squeeze(0).cpu()
+ sampling = {k: settings[k] for k in ("temperature", "top_k", "top_p")}
+ sampling["min_p"] = settings.get("min_p", 0.0)
+ sampling.update({k: advanced[k] for k in ("repetition_penalty", "repetition_window",
+ "token_run_penalty", "token_run_grace")})
+ seed = advanced.get("seed")
+ if seed is None:
+ seed = int(torch.randint(0, 2**31 - 1, ()).item())
+ params = SamplingParams(
+ temperature=1.0, top_k=-1, top_p=1.0, min_p=0.0,
+ repetition_penalty=1.0, max_tokens=int(max_new_tokens),
+ stop_token_ids=[self.token_map.speech_end_id, self.token_map.stop_id],
+ ignore_eos=True, detokenize=False, seed=int(seed),
+ extra_args={"kittenml": {"prompt": prompt, "token_map": asdict(self.token_map),
+ "sampling": sampling}})
+ return {"prompt_embeds": embeds}, params
+
+ def generate_tokens(self, prompt, speaker_embedding, settings, max_new_tokens, advanced):
+ request, params = self.prepare_request(prompt, speaker_embedding, settings,
+ max_new_tokens, advanced)
+ output = self.engine.generate([request], [params], use_tqdm=False)[0]
+ low, high = self.token_map.audio_id_base, self.token_map.audio_id_end
+ return [t - low for t in output.outputs[0].token_ids if low <= t < high]
diff --git a/kittenml/kittentts2/vllm/loader.py b/kittenml/kittentts2/vllm/loader.py
new file mode 100644
index 0000000..96b01f7
--- /dev/null
+++ b/kittenml/kittentts2/vllm/loader.py
@@ -0,0 +1,25 @@
+"""vLLM plugin: selected KittenTTS weights, without an intermediate export."""
+
+from vllm.model_executor.model_loader.base_loader import BaseModelLoader
+
+from ..weights import load_lm_weights
+
+
+class KittenModelLoader(BaseModelLoader):
+ def download_model(self, model_config):
+ # KittenTTS resolves the selected files before constructing the engine.
+ pass
+
+ def load_weights(self, model, model_config):
+ options = self.load_config.model_loader_extra_config
+ state = load_lm_weights(options["repo_dir"], options["config"], options["weights"])
+ loaded = model.load_weights((name, tensor) for name, tensor in state.items()
+ if not name.startswith("spk_proj."))
+ missing = set(dict(model.named_parameters())) - loaded
+ if missing:
+ raise ValueError(f"weights missing from the checkpoint: {sorted(missing)[:5]}")
+
+
+def register():
+ from vllm.model_executor.model_loader import register_model_loader
+ register_model_loader("kittenml")(KittenModelLoader)
diff --git a/kittenml/kittentts2/vllm/logits.py b/kittenml/kittentts2/vllm/logits.py
new file mode 100644
index 0000000..bcc0f4a
--- /dev/null
+++ b/kittenml/kittentts2/vllm/logits.py
@@ -0,0 +1,40 @@
+"""KittenTTS's complete sampling pipeline for vLLM Model Runner V2."""
+
+import torch
+from vllm.v1.worker.gpu.sample.logits_processor import LogitsProcessor
+
+from ..logits import build_logits_processors
+from ..tokens import TokenMap
+
+
+class KittenLogitsProcessor(LogitsProcessor):
+ def __init__(self, vllm_config, req_states):
+ self.req_states = req_states
+ self.requests = {}
+
+ def add_request(self, req_idx, sampling_params):
+ extra = (sampling_params.extra_args or {}).get("kittenml")
+ self.requests.pop(req_idx, None)
+ if extra is None:
+ return False
+ prompt = torch.tensor(extra["prompt"], device=self.req_states.device,
+ dtype=torch.long)
+ processors = build_logits_processors(
+ TokenMap(**extra["token_map"]), **extra["sampling"],
+ prompt_length=len(extra["prompt"]))
+ self.requests[req_idx] = (prompt, processors)
+ return True
+
+ def apply(self, logits, ctx):
+ # No speculative decoding: each row corresponds to one active request,
+ # and the host sequence lengths are exact (no per-token GPU readback).
+ for row, req_idx in enumerate(ctx.idx_mapping_np):
+ request = self.requests.get(int(req_idx))
+ if request is None:
+ continue
+ prompt, processors = request
+ length = int(ctx.seq_lens_upper_bound_np[row])
+ generated = self.req_states.all_token_ids.gpu[int(req_idx), len(prompt):length]
+ history = torch.cat((prompt, generated)).unsqueeze(0)
+ logits[row:row + 1] = processors(history, logits[row:row + 1])
+ return logits
diff --git a/kittenml/kittentts2/weights.py b/kittenml/kittentts2/weights.py
new file mode 100644
index 0000000..cc52c39
--- /dev/null
+++ b/kittenml/kittentts2/weights.py
@@ -0,0 +1,48 @@
+"""Read the selected checkpoint identically for Torch and vLLM."""
+
+from pathlib import Path
+
+
+def available_weights(config):
+ return ["full", *(name for name, key in (("packed", "lm_packed"), ("emb4", "lm_emb4"))
+ if config.get(key))]
+
+
+def weights_path(repo_dir, config, name):
+ if name == "full":
+ return None
+ key = {"packed": "lm_packed", "emb4": "lm_emb4"}.get(name)
+ if key is None:
+ raise ValueError(f"Unknown weights {name!r}. Choose from: {available_weights(config)}")
+ declared = config.get(key)
+ if not declared:
+ raise ValueError(f"this repository declares no {name!r} weights")
+ return Path(repo_dir) / declared
+
+
+def load_lm_weights(repo_dir, config, variant, lm_dir=None):
+ """Unpack on CPU without re-quantizing or writing an expanded checkpoint."""
+ from safetensors import safe_open
+ from safetensors.torch import load_file
+
+ path = weights_path(repo_dir, config, variant)
+ if path is not None:
+ if not path.is_file():
+ raise FileNotFoundError(f"{variant!r} weights not found: {path}")
+ if variant == "emb4":
+ from .tl2_emb4 import load_state_dict
+ return load_state_dict(path)
+ with safe_open(str(path), framework="pt") as handle:
+ metadata = handle.metadata() or {}
+ tensors = {k: handle.get_tensor(k) for k in handle.keys()}
+ from .ternary import unpack_state_dict
+ return unpack_state_dict(tensors, metadata)
+
+ lm_dir = Path(lm_dir) if lm_dir is not None else Path(repo_dir) / config.get("lm_dir", "lm")
+ state = {}
+ for shard in sorted(lm_dir.glob("model*.safetensors")):
+ if "-" not in shard.stem:
+ state.update(load_file(str(shard)))
+ if not state:
+ raise FileNotFoundError(f"no plain model weights in {lm_dir}")
+ return state
diff --git a/kittenml/kittentts_legacy/model.py b/kittenml/kittentts_legacy/model.py
index e3c6020..d92510a 100644
--- a/kittenml/kittentts_legacy/model.py
+++ b/kittenml/kittentts_legacy/model.py
@@ -25,7 +25,7 @@ class KittenTTSOnnx:
"""
def __init__(self, model_name="KittenML/kitten-tts-nano-0.8", cache_dir=None,
- backend=None, device=None, hf_token=None):
+ backend=None, device=None, hf_token=None, *, vllm_options=None):
"""Initialize an ONNX KittenTTS model.
Args:
@@ -35,6 +35,8 @@ class KittenTTSOnnx:
device: Unused here; accepted so both backends share a signature
hf_token: Unused here; accepted so both backends share a signature
"""
+ if vllm_options is not None:
+ raise ValueError("the vLLM backend requires a KittenTTS 2 model")
local_dir, repo_id, config = getattr(self, "_resolved", None) or \
resolve_repo(model_name, cache_dir)
self.model = build_onnx_model(config, local_dir, repo_id, cache_dir, backend)
diff --git a/pyproject.toml b/pyproject.toml
index a1323f4..d28404e 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "kittenml"
-version = "0.9.3"
+version = "0.9.4"
description = "Text-to-speech with voice cloning and expression control, plus ultra-lightweight models that run on CPU"
readme = "README.md"
requires-python = ">=3.10"
@@ -52,6 +52,12 @@ Homepage = "https://github.com/KittenML/KittenTTS"
Repository = "https://github.com/KittenML/KittenTTS"
Issues = "https://github.com/KittenML/KittenTTS/issues"
+[project.optional-dependencies]
+vllm = ["vllm==0.31.0; sys_platform == 'linux'"]
+
+[project.entry-points."vllm.general_plugins"]
+kittenml = "kittenml.kittentts2.vllm.loader:register"
+
[tool.setuptools.packages.find]
where = ["."]
include = ["kittenml*"]
diff --git a/setup.py b/setup.py
index 54b0d5d..34a57b8 100644
--- a/setup.py
+++ b/setup.py
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
setup(
name="kittenml",
- version="0.9.3",
+ version="0.9.4",
author="KittenML",
author_email="",
description="Text-to-speech with voice cloning and expression control, plus ultra-lightweight models that run on CPU",
diff --git a/tests/test_vllm.py b/tests/test_vllm.py
new file mode 100644
index 0000000..2a16ebd
--- /dev/null
+++ b/tests/test_vllm.py
@@ -0,0 +1,294 @@
+"""Small checkpoints and sampler parity; no GPU or model download required."""
+
+from dataclasses import asdict
+from types import SimpleNamespace
+from unittest import mock
+
+import numpy as np
+import pytest
+import torch
+from safetensors.torch import save_file
+
+from kittenml import KittenTTS
+from kittenml.kittentts2.ternary import pack_state_dict
+from kittenml.kittentts2.tokens import TokenMap
+from kittenml.kittentts2.weights import load_lm_weights
+
+
+def test_default_backend_keeps_native_loading(tmp_path):
+ from kittenml.kittentts2.model import KittenTTS2
+ config = {"token_map": asdict(TokenMap(5, 10, 1, 2, 3, 4, 0, 15))}
+ with mock.patch.object(KittenTTS2, "_load_language_model") as load, \
+ mock.patch("kittenml.kittentts2.model.S3Codec"), \
+ mock.patch("kittenml.kittentts2.model.SpeakerEncoder"), \
+ mock.patch("kittenml.kittentts2.model.ReferenceTranscriber"):
+ model = KittenTTS2(tmp_path, config, device="cpu")
+ load.assert_called_once_with()
+ assert model.backend == "torch" and model.device == "cpu"
+
+
+def test_selected_packed_weights_are_exact_and_full_ignores_packed_files(tmp_path):
+ lm = tmp_path / "lm"
+ lm.mkdir()
+ state = {"model.layer.weight": torch.arange(256).remainder(3).reshape(2, 128)
+ .sub(1).to(torch.bfloat16) * 0.125,
+ "spk_proj.0.bias": torch.tensor([0.7, -0.2], dtype=torch.bfloat16)}
+ save_file(state, lm / "model.safetensors")
+ tensors, metadata = pack_state_dict(state)
+ save_file(tensors, lm / "model-ternary.safetensors", metadata=metadata)
+ config = {"lm_packed": "lm/model-ternary.safetensors"}
+ for variant in ("packed", "full"):
+ result = load_lm_weights(tmp_path, config, variant)
+ assert result.keys() == state.keys()
+ assert all(torch.equal(result[name], tensor) for name, tensor in state.items())
+ with pytest.raises(ValueError, match="Unknown weights"):
+ load_lm_weights(tmp_path, config, "invalid")
+ with pytest.raises(ValueError, match="no 'emb4'"):
+ load_lm_weights(tmp_path, config, "emb4")
+
+
+def test_native_full_loader_preserves_explicit_checkpoint_directory(tmp_path):
+ from kittenml.kittentts2.model import KittenTTS2
+
+ lm_dir = tmp_path / "custom-lm"
+ lm_dir.mkdir()
+ expected = {"model.weight": torch.tensor([0.25, -0.5], dtype=torch.bfloat16)}
+ save_file(expected, lm_dir / "model.safetensors")
+ owner = SimpleNamespace(repo_dir=tmp_path, config={}, weights="full")
+ loaded = KittenTTS2._load_lm_weights(owner, lm_dir)
+ assert loaded.keys() == expected.keys()
+ torch.testing.assert_close(loaded["model.weight"], expected["model.weight"], rtol=0, atol=0)
+
+
+def test_shared_speaker_head_matches_native_prefill_and_leaves_decode_unchanged():
+ from kittenml.kittentts2.speaker import attach_speaker_head, load_speaker_head
+
+ class TinyLM(torch.nn.Module):
+ def __init__(self):
+ super().__init__()
+ self.config = SimpleNamespace(hidden_size=4)
+ self.embedding = torch.nn.Embedding(8, 4)
+
+ def get_input_embeddings(self):
+ return self.embedding
+
+ def forward(self, input_ids=None, inputs_embeds=None, cache_position=None):
+ return inputs_embeds
+
+ projection = torch.nn.Sequential(torch.nn.Linear(2, 4), torch.nn.LayerNorm(4))
+ state = {"spk_proj." + k: v for k, v in projection.state_dict().items()}
+ model = attach_speaker_head(TinyLM(), "unused", "cpu", torch.float32,
+ spk_dim=2, state=state)
+ head = load_speaker_head(4, "unused", "cpu", torch.float32, spk_dim=2, state=state)
+ speaker = torch.tensor([[0.5, -0.3]])
+ model._pending_speaker_embedding = speaker
+ ids = torch.tensor([[0, 3, 1]])
+ original = model.embedding(ids)
+ prefill = model(input_ids=ids, cache_position=torch.tensor([0]))
+ torch.testing.assert_close(prefill[:, 0], head(speaker))
+ torch.testing.assert_close(prefill[:, 1:], original[:, 1:])
+ decode = model(input_ids=ids, cache_position=torch.tensor([3]))
+ torch.testing.assert_close(decode, original)
+ with pytest.raises(ValueError, match="no spk_proj"):
+ load_speaker_head(4, "unused", "cpu", torch.float32, spk_dim=2, state={})
+
+
+def test_emb4_unpacking_is_independent_of_vllm_default_dtype():
+ from kittenml.kittentts2.tl2_emb4 import dequant_embedding
+ tensors = {
+ "emb4/tier_map": torch.zeros(2, dtype=torch.uint8),
+ "emb4/t0_codes": torch.zeros(1, dtype=torch.uint8),
+ "emb4/t0_centers": torch.tensor([[1., 2.], [3., 4.]], dtype=torch.float32),
+ "emb4/t0_s": torch.ones(2, dtype=torch.float32),
+ "emb4/t0_t": torch.ones(2, dtype=torch.float32),
+ }
+ metadata = {"V": "2", "D": "2", "Ks": "[2]"}
+ previous = torch.get_default_dtype()
+ try:
+ torch.set_default_dtype(torch.float32)
+ expected = dequant_embedding(tensors.__getitem__, metadata)
+ torch.set_default_dtype(torch.bfloat16)
+ actual = dequant_embedding(tensors.__getitem__, metadata)
+ finally:
+ torch.set_default_dtype(previous)
+ assert actual.dtype == torch.bfloat16 and torch.equal(actual, expected)
+
+
+def test_optional_backend_errors_before_model_download():
+ resolved = (None, "example/repo", {"type": "KITTEN2"})
+ with mock.patch("kittenml.get_model.resolve_repo", return_value=resolved), \
+ mock.patch("kittenml.kittentts2.vllm.backend.importlib.util.find_spec", return_value=None), \
+ mock.patch("kittenml.get_model.snapshot_download") as download:
+ with pytest.raises(ImportError, match=r"kittenml\[vllm\]"):
+ KittenTTS("example/repo", backend="vllm")
+ download.assert_not_called()
+ with mock.patch("kittenml.get_model.resolve_repo", return_value=(None, "legacy", {"type": "ONNX1"})):
+ with pytest.raises(ValueError, match="KittenTTS 2"):
+ KittenTTS("legacy", backend="vllm")
+
+
+def test_vllm_options_require_backend():
+ with mock.patch("kittenml.get_model.resolve_repo", return_value=(None, "model", {"type": "KITTEN2"})):
+ with pytest.raises(ValueError, match="requires backend"):
+ KittenTTS("model", vllm_options={})
+
+
+def test_explicit_default_options_do_not_break_legacy_models():
+ resolved = (None, "legacy", {"type": "ONNX1"})
+ with mock.patch("kittenml.get_model.resolve_repo", return_value=resolved), \
+ mock.patch("kittenml.kittentts_legacy.model.build_onnx_model") as build:
+ model = KittenTTS("legacy", vllm_options=None)
+ assert model.model is build.return_value
+ build.assert_called_once_with(resolved[2], None, "legacy", None, None)
+
+
+def test_environment_rejects_unsupported_devices_before_engine_start():
+ from kittenml.kittentts2.vllm.backend import validate_environment
+ with mock.patch("kittenml.kittentts2.vllm.backend.sys.platform", "win32"):
+ with pytest.raises(RuntimeError, match="Linux"):
+ validate_environment()
+ with mock.patch("kittenml.kittentts2.vllm.backend.importlib.util.find_spec", return_value=object()), \
+ mock.patch("torch.cuda.is_available", return_value=False):
+ with pytest.raises(RuntimeError, match="CUDA GPU"):
+ validate_environment()
+ with mock.patch("kittenml.kittentts2.vllm.backend.importlib.util.find_spec", return_value=object()), \
+ mock.patch("torch.cuda.is_available", return_value=True):
+ with pytest.raises(RuntimeError, match="CUDA GPU"):
+ validate_environment("cpu")
+ with pytest.raises(ValueError, match="CUDA_VISIBLE_DEVICES"):
+ validate_environment("cuda:1")
+
+
[email protected]("key, default, override", [
+ ("VLLM_USE_FLASHINFER_SAMPLER", "0", "1"),
+ ("VLLM_LOGGING_LEVEL", "WARNING", "INFO"),
+ ("TQDM_DISABLE", "1", "0"),
+])
+def test_engine_environment_preserves_user_choice_and_restores_after_failure(
+ monkeypatch, key, default, override):
+ import os
+ from kittenml.kittentts2.vllm.backend import _engine_environment
+ monkeypatch.delenv(key, raising=False)
+ with pytest.raises(RuntimeError):
+ with _engine_environment():
+ assert os.environ[key] == default
+ raise RuntimeError("engine failed")
+ assert key not in os.environ
+ monkeypatch.setenv(key, override)
+ with _engine_environment():
+ assert os.environ[key] == override
+ assert os.environ[key] == override
+
+
+def test_engine_receives_quiet_logging_config_without_changing_native_options(monkeypatch, tmp_path):
+ pytest.importorskip("vllm")
+ import os
+ from kittenml.kittentts2.vllm.backend import VLLMBackend
+
+ monkeypatch.delenv("VLLM_LOGGING_LEVEL", raising=False)
+ owner = SimpleNamespace(device="cuda", repo_dir=tmp_path, config={}, weights="packed",
+ token_map=None, _load_lm_weights=mock.Mock(return_value={
+ "model.embed_tokens.weight": torch.zeros(2, 4)}))
+
+ def create_engine(**kwargs):
+ assert kwargs["logging_config"].log_level == "WARNING"
+ assert os.environ["VLLM_LOGGING_LEVEL"] == "WARNING"
+ assert os.environ["TQDM_DISABLE"] == "1"
+ assert kwargs["enable_prompt_embeds"] and not kwargs["enable_prefix_caching"]
+ assert kwargs["model_loader_extra_config"]["weights"] == "packed"
+ return mock.sentinel.engine
+
+ with mock.patch("kittenml.kittentts2.vllm.backend.validate_environment"), \
+ mock.patch("transformers.AutoConfig.from_pretrained", return_value=SimpleNamespace(
+ model_type="qwen3", hidden_size=4)), \
+ mock.patch("transformers.AutoTokenizer.from_pretrained"), \
+ mock.patch("kittenml.kittentts2.speaker.load_speaker_head"), \
+ mock.patch("vllm.LLM", side_effect=create_engine):
+ backend = VLLMBackend(owner)
+ assert backend.engine is mock.sentinel.engine
+ assert "VLLM_LOGGING_LEVEL" not in os.environ
+
+
+def test_vllm_loader_filters_speaker_weights_and_rejects_missing_parameters():
+ pytest.importorskip("vllm")
+ from kittenml.kittentts2.vllm.loader import KittenModelLoader
+ loader = object.__new__(KittenModelLoader)
+ loader.load_config = SimpleNamespace(model_loader_extra_config={
+ "repo_dir": "unused", "config": {}, "weights": "packed"})
+
+ class TinyModel:
+ def load_weights(self, weights):
+ self.loaded = dict(weights)
+ return set(self.loaded)
+
+ def named_parameters(self):
+ return [("model.weight", None)]
+
+ model = TinyModel()
+ state = {"model.weight": torch.ones(2), "spk_proj.0.bias": torch.zeros(2)}
+ with mock.patch("kittenml.kittentts2.vllm.loader.load_lm_weights", return_value=state):
+ loader.load_weights(model, None)
+ assert set(model.loaded) == {"model.weight"}
+ with mock.patch("kittenml.kittentts2.vllm.loader.load_lm_weights", return_value={}):
+ with pytest.raises(ValueError, match="missing from the checkpoint"):
+ loader.load_weights(model, None)
+
+
[email protected]("window", [None, 3])
+def test_sampling_matches_native_when_requests_reorder_and_slots_are_reused(window):
+ pytest.importorskip("vllm")
+ from kittenml.kittentts2.logits import build_logits_processors
+ from kittenml.kittentts2.vllm.logits import KittenLogitsProcessor
+
+ tm = TokenMap(audio_id_base=5, num_audio_tokens=5000, speech_start_id=1,
+ speech_end_id=2, text_start_id=3, text_end_id=4, start_id=0, stop_id=5005)
+ sampling = dict(temperature=0.8, top_k=60, top_p=0.9, min_p=0.05,
+ repetition_penalty=1.1, repetition_window=window,
+ token_run_penalty=1.3, token_run_grace=3)
+ original = {0: [0, 3, 7, 1], 1: [0, 3, 8, 9, 1]}
+ history = {0: original[0] + [7, 7, 7, 7],
+ 1: original[1] + [tm.audio_id_base + 4299] * 4}
+ buf = torch.zeros(2, 16, dtype=torch.long)
+ # Embedding prompts may have placeholder IDs inside vLLM. The processor must
+ # restore the real prompt before applying full-history repetition penalties.
+ for slot in history:
+ buf[slot, len(original[slot]):len(history[slot])] = torch.tensor(history[slot][len(original[slot]):])
+ states = SimpleNamespace(device=torch.device("cpu"), all_token_ids=SimpleNamespace(gpu=buf))
+ processor = KittenLogitsProcessor(None, states)
+ for slot in original:
+ params = SimpleNamespace(extra_args={"kittenml": {"prompt": original[slot],
+ "token_map": asdict(tm), "sampling": sampling}})
+ assert processor.add_request(slot, params)
+ ctx = SimpleNamespace(idx_mapping_np=np.array([1, 0]),
+ seq_lens_upper_bound_np=np.array([len(history[1]), len(history[0])]))
+ scores = torch.randn(2, 5006, generator=torch.Generator().manual_seed(7))
+ expected = scores.clone()
+ for row, slot in enumerate([1, 0]):
+ native = build_logits_processors(tm, **sampling, prompt_length=len(original[slot]))
+ expected[row:row + 1] = native(torch.tensor([history[slot]]), expected[row:row + 1])
+ torch.testing.assert_close(processor.apply(scores, ctx), expected)
+ assert not processor.add_request(0, SimpleNamespace(extra_args=None))
+ assert 0 not in processor.requests
+
+
+def test_sampler_is_neutral_and_speaker_replaces_only_first_embedding():
+ pytest.importorskip("vllm")
+ from kittenml.kittentts2.vllm.backend import VLLMBackend
+
+ backend = object.__new__(VLLMBackend)
+ backend.max_model_len = 32
+ backend.token_map = TokenMap(5, 10, 1, 2, 3, 4, 0, 15)
+ backend.embedding = torch.arange(64, dtype=torch.float32).reshape(16, 4)
+ backend.spk_proj = torch.nn.Identity()
+ settings = dict(temperature=0.7, top_k=5, top_p=0.8, min_p=0.02)
+ advanced = dict(seed=42, repetition_penalty=1.2, repetition_window=3,
+ token_run_penalty=1.5, token_run_grace=4)
+ request, params = backend.prepare_request([0, 3, 1], torch.ones(1, 4), settings, 8, advanced)
+ torch.testing.assert_close(request["prompt_embeds"][0], torch.ones(4))
+ torch.testing.assert_close(request["prompt_embeds"][1:], backend.embedding[[3, 1]])
+ assert params.temperature == params.top_p == params.repetition_penalty == 1.0
+ assert params.top_k == -1 and params.min_p == 0.0 and params.seed == 42
+ assert params.stop_token_ids == [2, 15] and params.ignore_eos
+ assert params.extra_args["kittenml"]["sampling"]["temperature"] == 0.7
+ with pytest.raises(ValueError, match="token budget"):
+ backend.prepare_request([0, 3, 1], torch.ones(1, 4), settings, 30, advanced)