SOTA-Class TTS at Compact Scale
Python
Optimize SGLang Omni inference and Blackwell support
linux295650-ux1778e3ceab64
diff --git a/README.md b/README.mdindex 30c3bed..1730f33 100644--- a/README.md+++ b/README.md@@ -199,12 +199,36 @@ PORT=8010 \./sglang_omni/scripts/run_server.sh```+The default `fa3` attention backend is intended for Hopper GPUs such as H20+and H100. On consumer Blackwell GPUs such as RTX 5090, select FlashInfer for+the SGLang slow-AR path; the adapter then uses PyTorch SDPA for the short+fixed-cache fast head:++```bash+AUDIO8_TTS_ATTENTION_BACKEND=flashinfer \+CUDA_VISIBLE_DEVICES=0 \+SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \+MODEL="${MODEL}" \+./sglang_omni/scripts/run_server.sh+```+The defaults use model name `audio8/tts-0.6b`, BF16, one GPU, a `0.2` staticmemory fraction, and up to 32 running requests. The main runtime controls are`MODEL_NAME`, `AUDIO8_TTS_MEM_FRACTION_STATIC`,`AUDIO8_TTS_MAX_RUNNING_REQUESTS`, `AUDIO8_TTS_CHUNKED_PREFILL_SIZE`, and-`AUDIO8_TTS_DISABLE_CUDA_GRAPH`. Set `SGLANG_OMNI_SITE_PACKAGES` when the-runtime dependencies are installed in a separate site-packages directory.+`AUDIO8_TTS_DISABLE_CUDA_GRAPH`. `AUDIO8_TTS_ATTENTION_BACKEND` defaults to+`fa3`; set it to `flashinfer` for consumer Blackwell. Set+`SGLANG_OMNI_SITE_PACKAGES` when the runtime dependencies are installed in a+separate site-packages directory.++### Troubleshooting++- Install the distribution package that provides `libnuma` if `sgl_kernel`+ fails to import (for example, `numactl` or `libnuma1`).+- Put the CUDA toolkit `bin` directory on `PATH`. If `deep_gemm` cannot find+ `nvcc` during JIT compilation, also set `CUDA_PATH` to the toolkit root.+- Keep Transformers on the supported 4.x range (`>=4.57.0,<5`). Transformers+ 5.x can produce invalid all-zero codes for this custom-code model.### Call the APIdiff --git a/README_zh.md b/README_zh.mdindex 0478e09..4fb26b0 100644--- a/README_zh.md+++ b/README_zh.md@@ -185,11 +185,33 @@ PORT=8010 \./sglang_omni/scripts/run_server.sh```+默认的 `fa3` attention backend 适用于 H20、H100 等 Hopper GPU。在 RTX 5090 等消费级+Blackwell GPU 上,应为 SGLang Slow AR 路径选择 FlashInfer;此时适配器会在短序列固定+KV cache 的 Fast head 中使用 PyTorch SDPA:++```bash+AUDIO8_TTS_ATTENTION_BACKEND=flashinfer \+CUDA_VISIBLE_DEVICES=0 \+SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \+MODEL="${MODEL}" \+./sglang_omni/scripts/run_server.sh+```+默认配置使用模型名 `audio8/tts-0.6b`、BF16、单卡、`0.2` 静态显存比例和最多 32 个并发请求。主要运行参数包括 `MODEL_NAME`、`AUDIO8_TTS_MEM_FRACTION_STATIC`、`AUDIO8_TTS_MAX_RUNNING_REQUESTS`、`AUDIO8_TTS_CHUNKED_PREFILL_SIZE` 和-`AUDIO8_TTS_DISABLE_CUDA_GRAPH`。如果运行依赖安装在单独的 site-packages 目录中,请设置-`SGLANG_OMNI_SITE_PACKAGES`。+`AUDIO8_TTS_DISABLE_CUDA_GRAPH`。`AUDIO8_TTS_ATTENTION_BACKEND` 默认为 `fa3`;消费级+Blackwell GPU 应设置为 `flashinfer`。如果运行依赖安装在单独的 site-packages 目录中,请+设置 `SGLANG_OMNI_SITE_PACKAGES`。++### 常见问题++- 如果 `sgl_kernel` 导入失败,请安装系统中提供 `libnuma` 的软件包,例如 `numactl` 或+ `libnuma1`。+- CUDA toolkit 的 `bin` 目录需要加入 `PATH`。如果 `deep_gemm` JIT 编译时找不到+ `nvcc`,还需要将 `CUDA_PATH` 指向 CUDA toolkit 根目录。+- Transformers 应保持在支持的 4.x 范围(`>=4.57.0,<5`)。Transformers 5.x 可能使该+ custom-code 模型生成无效的全零 codes。### 调用 APIdiff --git a/sglang_omni/OPTIMIZATION_REPORT.md b/sglang_omni/OPTIMIZATION_REPORT.mdnew file mode 100644index 0000000..e2d95bd--- /dev/null+++ b/sglang_omni/OPTIMIZATION_REPORT.md@@ -0,0 +1,184 @@+# SGLang Omni Inference Optimization Report++## Overview++This update optimizes the Audio8 TTS SGLang Omni serving path for low-latency+single-stream inference while retaining the existing dynamic batching path. The+implementation was validated on an NVIDIA H20 with BF16 weights and CUDA Graph.+No CUDA, PyTorch, or model checkpoint changes are required.++## Code Changes++### Reduced semantic projection++Decode-time semantic logits are now computed only for the 4,096 valid semantic+tokens plus EOS. The previous implementation projected to all 155,776 tokenizer+entries and masked invalid entries afterward.++The reduced logits use a local candidate index during sampling and map the+selected result back to the original token ID. The full vocabulary projection+remains available before the Audio8 decode buffers are initialized.++### Fused fast decoder operations++- Fast decoder RMSNorm now uses the fused SGLang RMSNorm kernel. The wrapper+ flattens `[batch, 1, hidden]` to the two-dimensional input expected by the+ kernel and restores the original shape afterward.+- The fast decoder gate and up projections are merged into one linear layer+ after checkpoint loading. Existing `w1` and `w3` weights are concatenated+ once during service initialization.+- Checkpoint loading remains compatible with the original parameter names+ because fusion occurs only after all weights have been loaded.++### Optional Torch compilation++Two environment variables expose SGLang's existing `torch.compile` support:++```text+AUDIO8_TTS_ENABLE_TORCH_COMPILE=0|1+AUDIO8_TTS_TORCH_COMPILE_MAX_BS=1+```++Compilation is disabled by default. For single-stream deployment, compile only+batch size 1 and let batch sizes 2 and above use the existing CUDA Graph path.+The first startup performs compilation and kernel autotuning; later startups+reuse `TORCHINDUCTOR_CACHE_DIR`.++### Optional static greedy path++```text+AUDIO8_TTS_GREEDY_FASTPATH=0|1+```++When enabled, sampling is replaced by a direct argmax and all top-k, top-p, and+random sampling work is removed from the captured graph. This is a server-wide+mode and must only be enabled when every request is intended to use greedy+decoding. It overrides per-request sampling settings.++The CV3 quality evaluations in this report used+`AUDIO8_TTS_GREEDY_FASTPATH=0`.++### Portable Blackwell attention path++`AUDIO8_TTS_ATTENTION_BACKEND` controls both the SGLang slow-AR backend and the+fast-head cache implementation:++```text+AUDIO8_TTS_ATTENTION_BACKEND=fa3 # default for Hopper/H20+AUDIO8_TTS_ATTENTION_BACKEND=flashinfer # consumer Blackwell+```++The default path retains `flash_attn_with_kvcache`. When a non-`fa3` backend is+selected, the fast head writes K/V at the same fixed-cache positions in place+and uses SDPA with an explicit valid-prefix mask and GQA. Shapes remain static+for CUDA Graph capture. This avoids the Hopper-only FA3 custom op on `sm_120`+without changing the Hopper path.++For consumer Blackwell deployment, `sgl_kernel` also requires the system+`libnuma` library. The CUDA toolkit `bin` directory must be on `PATH`, and+`CUDA_PATH` may be required by `deep_gemm` JIT. Transformers must remain in the+supported `>=4.57.0,<5` range.++## Recommended Service Configuration++```bash+PATH=/path/to/ninja/bin:/usr/local/cuda/bin:${PATH} \+TORCHINDUCTOR_CACHE_DIR=/data/yumu/audio8_sglang_perf/torchinductor \+CUDA_VISIBLE_DEVICES=0 \+AUDIO8_TTS_MEM_FRACTION_STATIC=0.10 \+AUDIO8_TTS_MAX_RUNNING_REQUESTS=4 \+AUDIO8_TTS_ATTENTION_BACKEND=fa3 \+AUDIO8_TTS_ENABLE_TORCH_COMPILE=1 \+AUDIO8_TTS_TORCH_COMPILE_MAX_BS=1 \+AUDIO8_TTS_GREEDY_FASTPATH=0 \+sglang_omni/scripts/run_server.sh+```++`ninja` is needed only for first-time TorchInductor compilation. It does not+change the CUDA runtime.++## Performance++Environment:++```text+GPU: NVIDIA H20+PyTorch: 2.9.1+cu128+SGLang: 0.5.8+SGLang Omni: 0.1.0+Transformers: 4.57.1+Precision: BF16+CUDA Graph batch sizes: 1, 2, 4+```++The benchmark used one fixed Chinese request, 128 generated frames, greedy+decoding, and a 5.85-5.94 second WAV. Cold-start and compilation time were not+included.++| Path | Warm p50 latency | RTF | Relative speed |+|---|---:|---:|---:|+| Original PR service | 1.326 s | 0.227 | 1.00x |+| Optimized service | 0.691 s | 0.116 | 1.92x |++The server was tested with CUDA Graph capture for batch sizes 1, 2, and 4.+Ordinary TTS, reference-voice TTS, and four-request dynamic batching completed+without errors.++## CV3 Evaluation++All evaluations used the optimized service with reference audio, `top_p=0.8`,+`top_k=50`, and normal sampling. A total of 1,124 files were generated for the+main run and 124 files for the temperature comparison, with no generation+failures.++### Temperature 0.1++| Split | Metric | SIM |+|---|---:|---:|+| zh | CER 3.281 | 66.18 |+| en | WER 3.238 | 59.29 |+| hard_zh | CER 13.233 | 62.65 |+| hard_en | WER 6.941 | 57.69 |++Result directory:++```text+/data/yumu/tts_eval/CV3-Eval-main/results/audio8_sglang_optimized_cv3_20260802T0228Z+```++### Temperature 0.7 hard subsets++| Split | Metric | SIM |+|---|---:|---:|+| hard_zh | CER 10.927 | 63.40 |+| hard_en | WER 8.089 | 57.27 |++Result directory:++```text+/data/yumu/tts_eval/CV3-Eval-main/results/audio8_sglang_optimized_cv3_temp07_20260802T0251Z+```++Temperature 0.7 improves hard Chinese CER and similarity. Temperature 0.1 is+better for hard English WER and similarity. Recommended evaluation defaults:++```text+Chinese hard set: temperature=0.7+English hard set: temperature=0.1+top_p=0.8+top_k=50+```++## Validation Summary++- CUDA Graph capture succeeded for batch sizes 1, 2, and 4.+- Ordinary and reference-voice HTTP requests returned valid 44.1 kHz WAV files.+- Main CV3 run: 1,124/1,124 files generated, zero failures.+- Temperature 0.7 hard-set run: 124/124 files generated, zero failures.+- WER/CER and speaker-similarity scoring completed for every requested split.+- The portable SDPA cache path was checked on H20 against the full causal+ forward (`cosine=0.99999177` in BF16). CUDA Graph capture succeeded for batch+ sizes 1, 2, and 4, and an HTTP request returned a valid mono 44.1 kHz WAV.+- Consumer Blackwell `sm_120` execution and CUDA Graph capture were reported by+ the issue author; no `sm_120` GPU was available in the local validation+ environment.diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.pyindex ef04805..415262d 100644--- a/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py@@ -117,4 +117,13 @@ def make_server_args(model_path: str) -> Any:"disable_" + "cuda_graph",os.getenv("AUDIO8_TTS_DISABLE_CUDA_GRAPH", "0") == "1",)+ server_args.enable_torch_compile = (+ os.getenv("AUDIO8_TTS_ENABLE_TORCH_COMPILE", "0") == "1"+ )+ server_args.torch_compile_max_bs = int(+ os.getenv("AUDIO8_TTS_TORCH_COMPILE_MAX_BS", "1")+ )+ attention_backend = os.getenv("AUDIO8_TTS_ATTENTION_BACKEND")+ if attention_backend:+ server_args.attention_backend = attention_backendreturn server_argsdiff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/sglang_model.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/sglang_model.pyindex 986432f..c418539 100644--- a/sglang_omni/adapter/sglang_omni/models/audio8_tts/sglang_model.py+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/sglang_model.py@@ -4,6 +4,7 @@from __future__ import annotationsimport logging+import osfrom typing import Any, Iterable, Optional, Tupleimport torch@@ -40,16 +41,39 @@ def _audio8_fast_attention_with_cache(value: Tensor,cache_positions: Tensor,) -> Tensor:- return flash_attn_with_kvcache(- q=query,- k_cache=key_cache,- v_cache=value_cache,- k=key,- v=value,- cache_seqlens=cache_positions.contiguous(),- causal=True,- num_splits=0,+ attention_backend = os.getenv("AUDIO8_TTS_ATTENTION_BACKEND", "fa3").lower()+ if attention_backend == "fa3":+ return flash_attn_with_kvcache(+ q=query,+ k_cache=key_cache,+ v_cache=value_cache,+ k=key,+ v=value,+ cache_seqlens=cache_positions.contiguous(),+ causal=True,+ num_splits=0,+ )++ batch = query.shape[0]+ positions = cache_positions.to(torch.long)+ batch_indices = torch.arange(batch, device=query.device)+ key_cache[batch_indices, positions] = key[:, 0]+ value_cache[batch_indices, positions] = value[:, 0]++ sequence_length = key_cache.shape[1]+ valid_prefix = torch.arange(+ sequence_length,+ device=query.device,+ )[None] <= positions[:, None]+ output = F.scaled_dot_product_attention(+ query.transpose(1, 2),+ key_cache.transpose(1, 2),+ value_cache.transpose(1, 2),+ attn_mask=valid_prefix[:, None, None],+ is_causal=False,+ enable_gqa=True,)+ return output.transpose(1, 2).contiguous()@_audio8_fast_attention_with_cache.register_fake@@ -219,17 +243,14 @@ class Audio8DecoderLayer(nn.Module):return hidden_states, residual-class FastRMSNorm(nn.Module):+class FastRMSNorm(RMSNorm):def __init__(self, dim: int, eps: float) -> None:- super().__init__()- self.eps = float(eps)- self.weight = nn.Parameter(torch.ones(dim))+ super().__init__(dim, eps=eps)def forward(self, value: Tensor) -> Tensor:- normalized = value.float() * torch.rsqrt(- value.float().pow(2).mean(-1, keepdim=True) + self.eps- )- return normalized.to(value.dtype) * self.weight+ shape = value.shape+ normalized = super().forward(value.reshape(-1, shape[-1]))+ return normalized.view(shape)def _rope(length: int, head_dim: int, base: float, device: torch.device) -> Tensor:@@ -369,9 +390,31 @@ class FastFeedForward(nn.Module):self.w1 = nn.Linear(config.fast_dim, config.fast_intermediate_size, bias=False)self.w2 = nn.Linear(config.fast_intermediate_size, config.fast_dim, bias=False)self.w3 = nn.Linear(config.fast_dim, config.fast_intermediate_size, bias=False)+ self.gate_up: nn.Linear | None = None++ def fuse_gate_up(self) -> None:+ if self.gate_up is not None:+ return+ gate_up = nn.Linear(+ self.w1.in_features,+ self.w1.out_features * 2,+ bias=False,+ device=self.w1.weight.device,+ dtype=self.w1.weight.dtype,+ )+ with torch.no_grad():+ gate_up.weight.copy_(torch.cat((self.w1.weight, self.w3.weight), dim=0))+ self.gate_up = gate_up+ del self.w1+ del self.w3def forward(self, value: Tensor) -> Tensor:- return self.w2(F.silu(self.w1(value)) * self.w3(value))+ if self.gate_up is None:+ gate = self.w1(value)+ up = self.w3(value)+ else:+ gate, up = self.gate_up(value).chunk(2, dim=-1)+ return self.w2(F.silu(gate) * up)class FastDecoderLayer(nn.Module):@@ -535,6 +578,7 @@ class Audio8SGLangModel(nn.Module):persistent=False,)for layer in self.fast_layers:+ layer.feed_forward.fuse_gate_up()layer.attention.setup_audio8_cache(max_batch_size,config.num_codebooks + 1,@@ -562,6 +606,8 @@ class Audio8SGLangModel(nn.Module):top_k: Tensor,do_sample: Tensor,) -> Tensor:+ if os.getenv("AUDIO8_TTS_GREEDY_FASTPATH", "0") == "1":+ return scores.argmax(dim=-1)sorted_scores, sorted_indices = torch.sort(scores, descending=True, dim=-1)cumulative = torch.softmax(sorted_scores, dim=-1).cumsum(dim=-1)positions = torch.arange(scores.shape[-1], device=scores.device)[None]@@ -578,8 +624,8 @@ class Audio8SGLangModel(nn.Module):def _sample_semantic(self, logits: Tensor) -> Tensor:batch = logits.shape[0]- scores = logits.float() + self._semantic_bias- normal = self._sample(+ scores = logits.float()+ normal_index = self._sample(scores,self._temperature[:batch],self._top_p[:batch],@@ -590,13 +636,24 @@ class Audio8SGLangModel(nn.Module):self._temperature[:batch], self.config.ras_temperature)ras_top_p = torch.full_like(self._top_p[:batch], self.config.ras_top_p)- high = self._sample(+ high_index = self._sample(scores,ras_temperature,ras_top_p,self._top_k[:batch],self._do_sample[:batch],)+ semantic_begin = self.config.semantic_begin_id+ normal = torch.where(+ normal_index == 0,+ self.config.eos_token_id,+ semantic_begin + normal_index - 1,+ )+ high = torch.where(+ high_index == 0,+ self.config.eos_token_id,+ semantic_begin + high_index - 1,+ )repeated = ((self._previous_semantic[:batch] == normal[:, None])& self._previous_valid[:batch]@@ -606,6 +663,22 @@ class Audio8SGLangModel(nn.Module):)return torch.where(repeated & is_semantic, high, normal)+ def _semantic_logits(self, hidden_states: Tensor) -> Tensor:+ weight = self.embed_tokens.weight+ semantic_weight = weight[+ self.config.semantic_begin_id : self.config.semantic_end_id + 1+ ]+ eos_weight = weight[+ self.config.eos_token_id : self.config.eos_token_id + 1+ ]+ return torch.cat(+ (+ F.linear(hidden_states, eos_weight),+ F.linear(hidden_states, semantic_weight),+ ),+ dim=-1,+ )+def _clear_audio8_fast_caches(self) -> None:for layer in self.fast_layers:layer.attention.clear_audio8_cache()@@ -673,9 +746,11 @@ class Audio8SGLangModel(nn.Module):if forward_batch.forward_mode.is_extend():last_index = torch.cumsum(forward_batch.extend_seq_lens, dim=0) - 1hidden_states = hidden_states[last_index]- logits = F.linear(hidden_states, self.embed_tokens.weight)if self._decode_ready:+ logits = self._semantic_logits(hidden_states)self._decode_codebooks(logits, hidden_states)+ else:+ logits = F.linear(hidden_states, self.embed_tokens.weight)return LogitsProcessorOutput(next_token_logits=logits,hidden_states=hidden_states,