Skip to content

SOTA-Class TTS at Compact Scale

Python

Optimize SGLang Omni inference and Blackwell support

linux295650-ux1778e3ceab64
diff --git a/README.md b/README.md
index 30c3bed..1730f33 100644
--- a/README.md
+++ b/README.md
@@ -199,12 +199,36 @@ PORT=8010 \
./sglang_omni/scripts/run_server.sh
```
+The default `fa3` attention backend is intended for Hopper GPUs such as H20
+and H100. On consumer Blackwell GPUs such as RTX 5090, select FlashInfer for
+the SGLang slow-AR path; the adapter then uses PyTorch SDPA for the short
+fixed-cache fast head:
+
+```bash
+AUDIO8_TTS_ATTENTION_BACKEND=flashinfer \
+CUDA_VISIBLE_DEVICES=0 \
+SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \
+MODEL="${MODEL}" \
+./sglang_omni/scripts/run_server.sh
+```
+
The defaults use model name `audio8/tts-0.6b`, BF16, one GPU, a `0.2` static
memory fraction, and up to 32 running requests. The main runtime controls are
`MODEL_NAME`, `AUDIO8_TTS_MEM_FRACTION_STATIC`,
`AUDIO8_TTS_MAX_RUNNING_REQUESTS`, `AUDIO8_TTS_CHUNKED_PREFILL_SIZE`, and
-`AUDIO8_TTS_DISABLE_CUDA_GRAPH`. Set `SGLANG_OMNI_SITE_PACKAGES` when the
-runtime dependencies are installed in a separate site-packages directory.
+`AUDIO8_TTS_DISABLE_CUDA_GRAPH`. `AUDIO8_TTS_ATTENTION_BACKEND` defaults to
+`fa3`; set it to `flashinfer` for consumer Blackwell. Set
+`SGLANG_OMNI_SITE_PACKAGES` when the runtime dependencies are installed in a
+separate site-packages directory.
+
+### Troubleshooting
+
+- Install the distribution package that provides `libnuma` if `sgl_kernel`
+ fails to import (for example, `numactl` or `libnuma1`).
+- Put the CUDA toolkit `bin` directory on `PATH`. If `deep_gemm` cannot find
+ `nvcc` during JIT compilation, also set `CUDA_PATH` to the toolkit root.
+- Keep Transformers on the supported 4.x range (`>=4.57.0,<5`). Transformers
+ 5.x can produce invalid all-zero codes for this custom-code model.
### Call the API
diff --git a/README_zh.md b/README_zh.md
index 0478e09..4fb26b0 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -185,11 +185,33 @@ PORT=8010 \
./sglang_omni/scripts/run_server.sh
```
+默认的 `fa3` attention backend 适用于 H20、H100 等 Hopper GPU。在 RTX 5090 等消费级
+Blackwell GPU 上,应为 SGLang Slow AR 路径选择 FlashInfer;此时适配器会在短序列固定
+KV cache 的 Fast head 中使用 PyTorch SDPA:
+
+```bash
+AUDIO8_TTS_ATTENTION_BACKEND=flashinfer \
+CUDA_VISIBLE_DEVICES=0 \
+SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \
+MODEL="${MODEL}" \
+./sglang_omni/scripts/run_server.sh
+```
+
默认配置使用模型名 `audio8/tts-0.6b`、BF16、单卡、`0.2` 静态显存比例和最多 32 个并发
请求。主要运行参数包括 `MODEL_NAME`、`AUDIO8_TTS_MEM_FRACTION_STATIC`、
`AUDIO8_TTS_MAX_RUNNING_REQUESTS`、`AUDIO8_TTS_CHUNKED_PREFILL_SIZE` 和
-`AUDIO8_TTS_DISABLE_CUDA_GRAPH`。如果运行依赖安装在单独的 site-packages 目录中,请设置
-`SGLANG_OMNI_SITE_PACKAGES`。
+`AUDIO8_TTS_DISABLE_CUDA_GRAPH`。`AUDIO8_TTS_ATTENTION_BACKEND` 默认为 `fa3`;消费级
+Blackwell GPU 应设置为 `flashinfer`。如果运行依赖安装在单独的 site-packages 目录中,请
+设置 `SGLANG_OMNI_SITE_PACKAGES`。
+
+### 常见问题
+
+- 如果 `sgl_kernel` 导入失败,请安装系统中提供 `libnuma` 的软件包,例如 `numactl` 或
+ `libnuma1`。
+- CUDA toolkit 的 `bin` 目录需要加入 `PATH`。如果 `deep_gemm` JIT 编译时找不到
+ `nvcc`,还需要将 `CUDA_PATH` 指向 CUDA toolkit 根目录。
+- Transformers 应保持在支持的 4.x 范围(`>=4.57.0,<5`)。Transformers 5.x 可能使该
+ custom-code 模型生成无效的全零 codes。
### 调用 API
diff --git a/sglang_omni/OPTIMIZATION_REPORT.md b/sglang_omni/OPTIMIZATION_REPORT.md
new file mode 100644
index 0000000..e2d95bd
--- /dev/null
+++ b/sglang_omni/OPTIMIZATION_REPORT.md
@@ -0,0 +1,184 @@
+# SGLang Omni Inference Optimization Report
+
+## Overview
+
+This update optimizes the Audio8 TTS SGLang Omni serving path for low-latency
+single-stream inference while retaining the existing dynamic batching path. The
+implementation was validated on an NVIDIA H20 with BF16 weights and CUDA Graph.
+No CUDA, PyTorch, or model checkpoint changes are required.
+
+## Code Changes
+
+### Reduced semantic projection
+
+Decode-time semantic logits are now computed only for the 4,096 valid semantic
+tokens plus EOS. The previous implementation projected to all 155,776 tokenizer
+entries and masked invalid entries afterward.
+
+The reduced logits use a local candidate index during sampling and map the
+selected result back to the original token ID. The full vocabulary projection
+remains available before the Audio8 decode buffers are initialized.
+
+### Fused fast decoder operations
+
+- Fast decoder RMSNorm now uses the fused SGLang RMSNorm kernel. The wrapper
+ flattens `[batch, 1, hidden]` to the two-dimensional input expected by the
+ kernel and restores the original shape afterward.
+- The fast decoder gate and up projections are merged into one linear layer
+ after checkpoint loading. Existing `w1` and `w3` weights are concatenated
+ once during service initialization.
+- Checkpoint loading remains compatible with the original parameter names
+ because fusion occurs only after all weights have been loaded.
+
+### Optional Torch compilation
+
+Two environment variables expose SGLang's existing `torch.compile` support:
+
+```text
+AUDIO8_TTS_ENABLE_TORCH_COMPILE=0|1
+AUDIO8_TTS_TORCH_COMPILE_MAX_BS=1
+```
+
+Compilation is disabled by default. For single-stream deployment, compile only
+batch size 1 and let batch sizes 2 and above use the existing CUDA Graph path.
+The first startup performs compilation and kernel autotuning; later startups
+reuse `TORCHINDUCTOR_CACHE_DIR`.
+
+### Optional static greedy path
+
+```text
+AUDIO8_TTS_GREEDY_FASTPATH=0|1
+```
+
+When enabled, sampling is replaced by a direct argmax and all top-k, top-p, and
+random sampling work is removed from the captured graph. This is a server-wide
+mode and must only be enabled when every request is intended to use greedy
+decoding. It overrides per-request sampling settings.
+
+The CV3 quality evaluations in this report used
+`AUDIO8_TTS_GREEDY_FASTPATH=0`.
+
+### Portable Blackwell attention path
+
+`AUDIO8_TTS_ATTENTION_BACKEND` controls both the SGLang slow-AR backend and the
+fast-head cache implementation:
+
+```text
+AUDIO8_TTS_ATTENTION_BACKEND=fa3 # default for Hopper/H20
+AUDIO8_TTS_ATTENTION_BACKEND=flashinfer # consumer Blackwell
+```
+
+The default path retains `flash_attn_with_kvcache`. When a non-`fa3` backend is
+selected, the fast head writes K/V at the same fixed-cache positions in place
+and uses SDPA with an explicit valid-prefix mask and GQA. Shapes remain static
+for CUDA Graph capture. This avoids the Hopper-only FA3 custom op on `sm_120`
+without changing the Hopper path.
+
+For consumer Blackwell deployment, `sgl_kernel` also requires the system
+`libnuma` library. The CUDA toolkit `bin` directory must be on `PATH`, and
+`CUDA_PATH` may be required by `deep_gemm` JIT. Transformers must remain in the
+supported `>=4.57.0,<5` range.
+
+## Recommended Service Configuration
+
+```bash
+PATH=/path/to/ninja/bin:/usr/local/cuda/bin:${PATH} \
+TORCHINDUCTOR_CACHE_DIR=/data/yumu/audio8_sglang_perf/torchinductor \
+CUDA_VISIBLE_DEVICES=0 \
+AUDIO8_TTS_MEM_FRACTION_STATIC=0.10 \
+AUDIO8_TTS_MAX_RUNNING_REQUESTS=4 \
+AUDIO8_TTS_ATTENTION_BACKEND=fa3 \
+AUDIO8_TTS_ENABLE_TORCH_COMPILE=1 \
+AUDIO8_TTS_TORCH_COMPILE_MAX_BS=1 \
+AUDIO8_TTS_GREEDY_FASTPATH=0 \
+sglang_omni/scripts/run_server.sh
+```
+
+`ninja` is needed only for first-time TorchInductor compilation. It does not
+change the CUDA runtime.
+
+## Performance
+
+Environment:
+
+```text
+GPU: NVIDIA H20
+PyTorch: 2.9.1+cu128
+SGLang: 0.5.8
+SGLang Omni: 0.1.0
+Transformers: 4.57.1
+Precision: BF16
+CUDA Graph batch sizes: 1, 2, 4
+```
+
+The benchmark used one fixed Chinese request, 128 generated frames, greedy
+decoding, and a 5.85-5.94 second WAV. Cold-start and compilation time were not
+included.
+
+| Path | Warm p50 latency | RTF | Relative speed |
+|---|---:|---:|---:|
+| Original PR service | 1.326 s | 0.227 | 1.00x |
+| Optimized service | 0.691 s | 0.116 | 1.92x |
+
+The server was tested with CUDA Graph capture for batch sizes 1, 2, and 4.
+Ordinary TTS, reference-voice TTS, and four-request dynamic batching completed
+without errors.
+
+## CV3 Evaluation
+
+All evaluations used the optimized service with reference audio, `top_p=0.8`,
+`top_k=50`, and normal sampling. A total of 1,124 files were generated for the
+main run and 124 files for the temperature comparison, with no generation
+failures.
+
+### Temperature 0.1
+
+| Split | Metric | SIM |
+|---|---:|---:|
+| zh | CER 3.281 | 66.18 |
+| en | WER 3.238 | 59.29 |
+| hard_zh | CER 13.233 | 62.65 |
+| hard_en | WER 6.941 | 57.69 |
+
+Result directory:
+
+```text
+/data/yumu/tts_eval/CV3-Eval-main/results/audio8_sglang_optimized_cv3_20260802T0228Z
+```
+
+### Temperature 0.7 hard subsets
+
+| Split | Metric | SIM |
+|---|---:|---:|
+| hard_zh | CER 10.927 | 63.40 |
+| hard_en | WER 8.089 | 57.27 |
+
+Result directory:
+
+```text
+/data/yumu/tts_eval/CV3-Eval-main/results/audio8_sglang_optimized_cv3_temp07_20260802T0251Z
+```
+
+Temperature 0.7 improves hard Chinese CER and similarity. Temperature 0.1 is
+better for hard English WER and similarity. Recommended evaluation defaults:
+
+```text
+Chinese hard set: temperature=0.7
+English hard set: temperature=0.1
+top_p=0.8
+top_k=50
+```
+
+## Validation Summary
+
+- CUDA Graph capture succeeded for batch sizes 1, 2, and 4.
+- Ordinary and reference-voice HTTP requests returned valid 44.1 kHz WAV files.
+- Main CV3 run: 1,124/1,124 files generated, zero failures.
+- Temperature 0.7 hard-set run: 124/124 files generated, zero failures.
+- WER/CER and speaker-similarity scoring completed for every requested split.
+- The portable SDPA cache path was checked on H20 against the full causal
+ forward (`cosine=0.99999177` in BF16). CUDA Graph capture succeeded for batch
+ sizes 1, 2, and 4, and an HTTP request returned a valid mono 44.1 kHz WAV.
+- Consumer Blackwell `sm_120` execution and CUDA Graph capture were reported by
+ the issue author; no `sm_120` GPU was available in the local validation
+ environment.
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py
index ef04805..415262d 100644
--- a/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py
@@ -117,4 +117,13 @@ def make_server_args(model_path: str) -> Any:
"disable_" + "cuda_graph",
os.getenv("AUDIO8_TTS_DISABLE_CUDA_GRAPH", "0") == "1",
)
+ server_args.enable_torch_compile = (
+ os.getenv("AUDIO8_TTS_ENABLE_TORCH_COMPILE", "0") == "1"
+ )
+ server_args.torch_compile_max_bs = int(
+ os.getenv("AUDIO8_TTS_TORCH_COMPILE_MAX_BS", "1")
+ )
+ attention_backend = os.getenv("AUDIO8_TTS_ATTENTION_BACKEND")
+ if attention_backend:
+ server_args.attention_backend = attention_backend
return server_args
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/sglang_model.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/sglang_model.py
index 986432f..c418539 100644
--- a/sglang_omni/adapter/sglang_omni/models/audio8_tts/sglang_model.py
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/sglang_model.py
@@ -4,6 +4,7 @@
from __future__ import annotations
import logging
+import os
from typing import Any, Iterable, Optional, Tuple
import torch
@@ -40,16 +41,39 @@ def _audio8_fast_attention_with_cache(
value: Tensor,
cache_positions: Tensor,
) -> Tensor:
- return flash_attn_with_kvcache(
- q=query,
- k_cache=key_cache,
- v_cache=value_cache,
- k=key,
- v=value,
- cache_seqlens=cache_positions.contiguous(),
- causal=True,
- num_splits=0,
+ attention_backend = os.getenv("AUDIO8_TTS_ATTENTION_BACKEND", "fa3").lower()
+ if attention_backend == "fa3":
+ return flash_attn_with_kvcache(
+ q=query,
+ k_cache=key_cache,
+ v_cache=value_cache,
+ k=key,
+ v=value,
+ cache_seqlens=cache_positions.contiguous(),
+ causal=True,
+ num_splits=0,
+ )
+
+ batch = query.shape[0]
+ positions = cache_positions.to(torch.long)
+ batch_indices = torch.arange(batch, device=query.device)
+ key_cache[batch_indices, positions] = key[:, 0]
+ value_cache[batch_indices, positions] = value[:, 0]
+
+ sequence_length = key_cache.shape[1]
+ valid_prefix = torch.arange(
+ sequence_length,
+ device=query.device,
+ )[None] <= positions[:, None]
+ output = F.scaled_dot_product_attention(
+ query.transpose(1, 2),
+ key_cache.transpose(1, 2),
+ value_cache.transpose(1, 2),
+ attn_mask=valid_prefix[:, None, None],
+ is_causal=False,
+ enable_gqa=True,
)
+ return output.transpose(1, 2).contiguous()
@_audio8_fast_attention_with_cache.register_fake
@@ -219,17 +243,14 @@ class Audio8DecoderLayer(nn.Module):
return hidden_states, residual
-class FastRMSNorm(nn.Module):
+class FastRMSNorm(RMSNorm):
def __init__(self, dim: int, eps: float) -> None:
- super().__init__()
- self.eps = float(eps)
- self.weight = nn.Parameter(torch.ones(dim))
+ super().__init__(dim, eps=eps)
def forward(self, value: Tensor) -> Tensor:
- normalized = value.float() * torch.rsqrt(
- value.float().pow(2).mean(-1, keepdim=True) + self.eps
- )
- return normalized.to(value.dtype) * self.weight
+ shape = value.shape
+ normalized = super().forward(value.reshape(-1, shape[-1]))
+ return normalized.view(shape)
def _rope(length: int, head_dim: int, base: float, device: torch.device) -> Tensor:
@@ -369,9 +390,31 @@ class FastFeedForward(nn.Module):
self.w1 = nn.Linear(config.fast_dim, config.fast_intermediate_size, bias=False)
self.w2 = nn.Linear(config.fast_intermediate_size, config.fast_dim, bias=False)
self.w3 = nn.Linear(config.fast_dim, config.fast_intermediate_size, bias=False)
+ self.gate_up: nn.Linear | None = None
+
+ def fuse_gate_up(self) -> None:
+ if self.gate_up is not None:
+ return
+ gate_up = nn.Linear(
+ self.w1.in_features,
+ self.w1.out_features * 2,
+ bias=False,
+ device=self.w1.weight.device,
+ dtype=self.w1.weight.dtype,
+ )
+ with torch.no_grad():
+ gate_up.weight.copy_(torch.cat((self.w1.weight, self.w3.weight), dim=0))
+ self.gate_up = gate_up
+ del self.w1
+ del self.w3
def forward(self, value: Tensor) -> Tensor:
- return self.w2(F.silu(self.w1(value)) * self.w3(value))
+ if self.gate_up is None:
+ gate = self.w1(value)
+ up = self.w3(value)
+ else:
+ gate, up = self.gate_up(value).chunk(2, dim=-1)
+ return self.w2(F.silu(gate) * up)
class FastDecoderLayer(nn.Module):
@@ -535,6 +578,7 @@ class Audio8SGLangModel(nn.Module):
persistent=False,
)
for layer in self.fast_layers:
+ layer.feed_forward.fuse_gate_up()
layer.attention.setup_audio8_cache(
max_batch_size,
config.num_codebooks + 1,
@@ -562,6 +606,8 @@ class Audio8SGLangModel(nn.Module):
top_k: Tensor,
do_sample: Tensor,
) -> Tensor:
+ if os.getenv("AUDIO8_TTS_GREEDY_FASTPATH", "0") == "1":
+ return scores.argmax(dim=-1)
sorted_scores, sorted_indices = torch.sort(scores, descending=True, dim=-1)
cumulative = torch.softmax(sorted_scores, dim=-1).cumsum(dim=-1)
positions = torch.arange(scores.shape[-1], device=scores.device)[None]
@@ -578,8 +624,8 @@ class Audio8SGLangModel(nn.Module):
def _sample_semantic(self, logits: Tensor) -> Tensor:
batch = logits.shape[0]
- scores = logits.float() + self._semantic_bias
- normal = self._sample(
+ scores = logits.float()
+ normal_index = self._sample(
scores,
self._temperature[:batch],
self._top_p[:batch],
@@ -590,13 +636,24 @@ class Audio8SGLangModel(nn.Module):
self._temperature[:batch], self.config.ras_temperature
)
ras_top_p = torch.full_like(self._top_p[:batch], self.config.ras_top_p)
- high = self._sample(
+ high_index = self._sample(
scores,
ras_temperature,
ras_top_p,
self._top_k[:batch],
self._do_sample[:batch],
)
+ semantic_begin = self.config.semantic_begin_id
+ normal = torch.where(
+ normal_index == 0,
+ self.config.eos_token_id,
+ semantic_begin + normal_index - 1,
+ )
+ high = torch.where(
+ high_index == 0,
+ self.config.eos_token_id,
+ semantic_begin + high_index - 1,
+ )
repeated = (
(self._previous_semantic[:batch] == normal[:, None])
& self._previous_valid[:batch]
@@ -606,6 +663,22 @@ class Audio8SGLangModel(nn.Module):
)
return torch.where(repeated & is_semantic, high, normal)
+ def _semantic_logits(self, hidden_states: Tensor) -> Tensor:
+ weight = self.embed_tokens.weight
+ semantic_weight = weight[
+ self.config.semantic_begin_id : self.config.semantic_end_id + 1
+ ]
+ eos_weight = weight[
+ self.config.eos_token_id : self.config.eos_token_id + 1
+ ]
+ return torch.cat(
+ (
+ F.linear(hidden_states, eos_weight),
+ F.linear(hidden_states, semantic_weight),
+ ),
+ dim=-1,
+ )
+
def _clear_audio8_fast_caches(self) -> None:
for layer in self.fast_layers:
layer.attention.clear_audio8_cache()
@@ -673,9 +746,11 @@ class Audio8SGLangModel(nn.Module):
if forward_batch.forward_mode.is_extend():
last_index = torch.cumsum(forward_batch.extend_seq_lens, dim=0) - 1
hidden_states = hidden_states[last_index]
- logits = F.linear(hidden_states, self.embed_tokens.weight)
if self._decode_ready:
+ logits = self._semantic_logits(hidden_states)
self._decode_codebooks(logits, hidden_states)
+ else:
+ logits = F.linear(hidden_states, self.embed_tokens.weight)
return LogitsProcessorOutput(
next_token_logits=logits,
hidden_states=hidden_states,