Open-source State-of-the-art TTS model which runs on a CPU π»
Running with vLLM
The optional vllm backend runs KittenTTS 2's language model on the GPU. Voices, cloning,
expression controls and audio decoding use the same API. These instructions use vLLM 0.31.0.
Use the README example on Linux or NVIDIA WSL2 with
NVIDIA driver 580 or newer, compute capability 7.5 or newer and at least 6 GB of VRAM.
vLLM also needs a C compiler and Python development headers
(gcc and python3-dev on Ubuntu); see its requirements.
For other setups, expand the relevant section below. Native Windows is not supported by vLLM.
The first start downloads weights and compiles kernels. Reuse the loaded model across requests. See benchmarks for performance comparisons.
Options
Pass options when constructing the model, for example vllm_options={"max_model_len": 2048}.
| Option | Default | What it is for |
|---|---|---|
gpu_memory_utilization |
0.5 |
Share of GPU memory for the language model and cache; leave room for audio decoding. |
max_model_len |
4096 |
Maximum tokens for the reference prompt, input text and generated audio together. |
max_num_seqs |
8 |
Maximum concurrent requests. |
max_num_batched_tokens |
vLLM's default | Tokens processed per step; lower values use less working memory. |
kv_cache_memory_bytes |
not set | Fixed cache size instead of sizing it from gpu_memory_utilization. |
enforce_eager |
False |
Disable CUDA graphs; normally leave this off. |
quantization |
not set; "w4a16" on RTX 20 / GTX 16 |
Use "w4a16" for 4-bit weights, or None for dense weights. |
dtype |
"bfloat16"; "float16" on compute capability 7.x |
Language-model precision. "float32" needs more memory and cannot use 4-bit weights. |
Unknown options raise an error. Select a GPU with CUDA_VISIBLE_DEVICES; set
VLLM_LOGGING_LEVEL=INFO to show engine logs.
Small GPUs
On GPUs with 8 GB or less, and on compute capability 7.x GPUs, KittenTTS adjusts unset options
if the defaults do not fit. It can select 4-bit weights, reduce the cache or shorten the
context, and logs the chosen settings. Explicit options are preserved; setting
gpu_memory_utilization or kv_cache_memory_bytes disables automatic fitting.
This supports 6β8 GB GPUs with enough free memory, but can limit concurrent requests and maximum request length. If even the smallest settings do not fit, loading stops with the required free-memory amount. 4 GB cards are insufficient for this vLLM path.
For voice cloning, pass reference_text to skip transcription. Otherwise Whisper uses the
CPU when GPU memory is low, which takes longer and needs additional host RAM.
4-bit weights
Set vllm_options={"quantization": "w4a16"} to reduce language-model weight memory.
With bf16, the weight values are preserved; float16 also rounds them. Generated audio can
differ because the kernels differ.
By default, vLLM gives the saved memory to its cache. To reduce total memory use, also lower
gpu_memory_utilization or set kv_cache_memory_bytes. The separate weights="emb4" option
reduces download size, not GPU memory, and cannot be used with float16.
Older NVIDIA drivers
CUDA 12.9 installation for drivers older than 580
For GPUs with compute capability 7.5 or newer, use the CUDA 12.9 builds if your driver supports CUDA 12 but not CUDA 13. Install uv and the compiler prerequisites above, then open a new shell with no active Python environment:
uv venv --seed --python 3.12 ~/kitten-env-cu129
source ~/kitten-env-cu129/bin/activate
uv pip install --torch-backend cu129 "kittenml[vllm]" \
https://github.com/vllm-project/vllm/releases/download/v0.31.0/vllm-0.31.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl
uv pip uninstall torchcodec
The last line removes an unused optional decoder that can conflict with this CUDA build.
Run the README's vllm_hello.py in this environment. Activate ~/kitten-env-cu129 again
in each new terminal.
Older GPUs (RTX 20 and GTX 16 series)
Supported, not optimal. These GPUs have no native bfloat16, so KittenTTS selects rounded float16 with 4-bit weights. Use the normal install or the older-driver instructions above; no extra precision option is needed. Audio can differ from newer GPUs.
At least 6 GB of VRAM is needed. To use dense float16, set {"quantization": None};
{"dtype": "float32"} is an alternative that needs more memory. GPUs below compute
capability 7.0, including GTX 10, cannot use this backend; use device="cpu" with the
default backend instead.
V100 (CUDA 12.6 source build)
Build vLLM for V100
Supported, not optimal. V100 uses rounded float16 with dense weights. It cannot use bfloat16 or the 4-bit kernel. The standard CUDA 12.9/13 wheels and KittenTTS Docker image cannot run on it.
Use Linux x86_64 with Python 3.12, Git, GCC/G++ and Python development headers. Install the
CUDA 12.6 development toolkit
first; nvcc --version must report 12.6. Then, in a new shell:
python3 -m venv ~/kitten-env-v100
source ~/kitten-env-v100/bin/activate
python -m pip install --upgrade pip uv
nvcc --version
uv pip install --torch-backend cu126 torch==2.13.0 torchaudio==2.11.0 torchvision==0.28.0
git clone --depth 1 --branch v0.31.0 https://github.com/vllm-project/vllm.git ~/vllm-v100
cd ~/vllm-v100
uv pip install --torch-backend cu126 -r requirements/build/cuda.txt
TORCH_CUDA_ARCH_LIST=7.0 MAX_JOBS=8 NVCC_THREADS=2 VLLM_USE_PRECOMPILED_RUST=1 uv pip install --torch-backend cu126 --no-build-isolation .
uv pip install --torch-backend cu126 kittenml
uv pip uninstall torchcodec
cd ~
Activate ~/kitten-env-v100 in each new terminal. Save this as v100_hello.py and run
python v100_hello.py; it leaves out the README example's unsupported 4-bit option:
from kittenml import KittenTTS
import soundfile as sf
if __name__ == "__main__":
model = KittenTTS("KittenML/kitten-tts-2", backend="vllm")
audio = model.generate("Hello from KittenTTS!", voice="Bruno")
sf.write("v100_output.wav", audio, model.sample_rate)
print("Saved v100_output.wav")
AMD GPUs (ROCm)
Radeon RX 7000 (RX 7700 and up) and RX 9000 series use a separate ROCm installation. The WSL2 application path is tested; native Linux remains untested. RX 7600 and older Radeon GPUs are not supported by this vLLM build.
Windows 11 with WSL2
Use Ubuntu 24.04 in WSL2 with Python 3.12, a C compiler and Python development headers.
Follow AMD's WSL2 setup
for the Windows driver, and ROCDXG setup
for ROCm and its WSL2 bridge. Use ROCm 7.2.4 and librocdxg 1.2.2 with the package versions
below. Do not install amdgpu-dkms inside WSL2. Windows-side setup was not part of our testing.
Once rocminfo lists your GPU, create a fresh environment in Ubuntu:
python3 -m venv ~/kitten-env-rocm
source ~/kitten-env-rocm/bin/activate
pip install uv
uv pip install "kittenml[vllm]" --extra-index-url https://wheels.vllm.ai/rocm/0.31.0/rocm723
Save the README's vllm_hello.py. In each terminal, activate the environment and set the
WSL2 compatibility variables before running it:
source ~/kitten-env-rocm/bin/activate
export HSA_ENABLE_DXG_DETECTION=1
export ROCPROFILER_REGISTER_ENABLED=0
python vllm_hello.py
Native Linux (untested)
Set up the host with AMD's Linux driver instructions
and Docker. The GPU must be accessible through
/dev/kfd and /dev/dri. Start vLLM's ROCm container, then install KittenTTS inside it:
docker run -it --device /dev/kfd --device /dev/dri --group-add video --ipc=host \
--cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--entrypoint bash vllm/vllm-openai-rocm:v0.31.0
pip install "kittenml[vllm]"
Use backend="vllm" in your script. These device flags are for native Linux, not WSL2.
Output differences
The same seed does not guarantee identical audio across GPU types, backends or precision
settings. Cloning can also differ when CPU transcription uses a different Whisper model;
provide reference_text to skip that step.
Benchmarks
Times vary with hardware, settings and text; startup is excluded unless stated.
PyTorch and vLLM
One RTX 3060 (12 GB), driver 570.195.03 (CUDA 12.8), Ubuntu 24.04, Python 3.12, nothing else on the
GPU. PyTorch GPU is the default backend (PyTorch 2.14.1 CUDA 12.6 build, Transformers 5.20.2); the
vLLM columns use vLLM 0.31.0 (CUDA 12.9 build, PyTorch 2.13.0, Transformers 5.17.0) with default
vllm_options: "vLLM bf16" is without quantization, "vLLM 4-bit" adds "quantization": "w4a16"
(4-bit weights). Each time is the median of five generate() calls after two
warmups (weights="packed", default decoder, stable preset, model loading excluded) on 61, 226 and
680 characters of text (the last in two chunks).
| Input / voice | PyTorch GPU | vLLM bf16 | vLLM 4-bit | Speedup bf16 vs PyTorch | Speedup 4-bit vs PyTorch |
|---|---|---|---|---|---|
| Short / Bruno | 4.44 s | 1.72 s | 1.18 s | 2.59x | 3.77x |
| Short / Bella | 6.14 s | 2.67 s | 1.62 s | 2.30x | 3.78x |
| Medium / Bruno | 14.74 s | 5.12 s | 3.43 s | 2.88x | 4.30x |
| Medium / Bella | 20.63 s | 7.01 s | 4.63 s | 2.94x | 4.46x |
| Long / Bruno | 45.20 s | 16.10 s | 10.38 s | 2.81x | 4.35x |
| Long / Bella | 53.15 s | 18.96 s | 12.24 s | 2.80x | 4.34x |
| RTX 3060 (12 GB) | PyTorch GPU | vLLM bf16 | vLLM 4-bit |
|---|---|---|---|
| Synthesis time per second of audio | 1.17-1.19 s | 0.41-0.45 s | 0.26-0.30 s |
| Peak GPU memory, whole card | 6,901 MiB | 9,026 MiB | 9,890 MiB |
| Start-up with files already downloaded (very first start) | 9.8 s (11.2 s) | 44.0 s (145.4 s) | 50.3 s (140.7 s) |
Per second of audio the speedups are 2.64-2.85x (bf16) and 3.94-4.51x (4-bit). Both vLLM columns give vLLM the same share of the card, so 4-bit gets a larger cache (37,120 tokens instead of 18,512); its higher whole-card peak was in the main process, which decodes the audio, not in vLLM's own process. Whole-card peaks include about 374 MiB the GPU reports as used when idle. A full repeat of every column was within -0.9% to +3.4%.
Older NVIDIA GPUs
These modes used the same model revision and 72 requests: three voices, six texts, two seeds, streaming and complete audio. RTF is generation time divided by audio length; lower is faster. First audio is the median for short streaming requests. Memory is whole-card peak use.
| GPU and mode | Word error rate | Median RTF | First audio | Peak GPU memory |
|---|---|---|---|---|
| RTX 3060 12 GB, bf16 reference | 0.25% | 0.45 | 1.52 s | 10,466 MiB |
| RTX 2060 12 GB, rounded float16 | 0.25% | 0.48 | 1.60 s | 10,144 MiB |
| RTX 2060 12 GB, rounded float16 + 4-bit (chosen) | 0.25% | 0.33 | 1.15 s | 9,672 MiB |
| RTX 2060 12 GB, float32 | 0.25% | 0.79 | 2.55 s | 11,640 MiB |
| GTX 1660 Ti 6 GB, rounded float16 + 4-bit | 0.25% | 1.12 | 3.75 s | 5,608 MiB |
The float32 RTX 2060 run used gpu_memory_utilization=0.74 and
max_num_batched_tokens=2048. The GTX 1660 Ti used automatic memory fitting.
| V100 32 GB mode | Word error rate | Median RTF | First audio | Peak GPU memory |
|---|---|---|---|---|
| Rounded float16 (chosen) | 0.25% | 0.25 | 0.82 s | 20,747 MiB |
| Float32 | 0.25% | 0.36 | 1.21 s | 20,953 MiB |
V100 used the CUDA 12.6 source build and default memory settings. All modes in these two tables had eight word errors in 3,248 words, no detected dropped sentences, and no silent or non-finite audio. These are transcription measurements; human listening remains pending.
AMD Radeon under WSL2
Each mode used 72 requests: three voices, six texts, two seeds, streaming and complete audio. The RX 7900 XTX and RX 9070 XT used vLLM's ROCm image under WSL2; the RX 9060 XT and RX 7800 XT used the WSL2 Python environment. These are not native Linux results.
| GPU | bf16: seconds per second of audio | 4-bit: seconds per second of audio | Word error rate: bf16 / 4-bit |
|---|---|---|---|
| RX 7900 XTX | 0.24 | 0.18 | 0.31% / 0.31% |
| RX 9070 XT | 0.28 | 0.21 | 0.31% / 0.31% |
| RX 9060 XT | 0.47 | 0.32 | 0.25% / 0.62% |
| RX 7800 XT | 0.36 | 0.26 | 0.31% / 0.31% |
No request lost words on these cards. The RX 7900 XTX was measured before the long-text decoder fix; the RX 9070 XT includes it. Transcription scores do not establish identical sound across GPUs or replace listening checks.