SOTA-Class TTS at Compact Scale
Python
Enable Torch compile for batched inference
linux295650-uxc6e10eb2849f
diff --git a/README.md b/README.mdindex 2bbc1ed..bcde98e 100644--- a/README.md+++ b/README.md@@ -208,6 +208,8 @@ SGLANG_OMNI_PACKAGE="$(python3 -c 'import importlib.util, pathlib; s=importlib.uCUDA_VISIBLE_DEVICES=0 \SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \MODEL="${MODEL}" \+AUDIO8_TTS_ENABLE_TORCH_COMPILE=1 \+AUDIO8_TTS_TORCH_COMPILE_MAX_BS=4 \HOST=0.0.0.0 \PORT=8010 \./sglang_omni/scripts/run_server.sh@@ -230,8 +232,10 @@ The defaults use model name `audio8/tts-0.6b`, BF16, one GPU, a `0.2` staticmemory fraction, and up to 32 running requests. The main runtime controls are`MODEL_NAME`, `AUDIO8_TTS_MEM_FRACTION_STATIC`,`AUDIO8_TTS_MAX_RUNNING_REQUESTS`, `AUDIO8_TTS_CHUNKED_PREFILL_SIZE`, and-`AUDIO8_TTS_DISABLE_CUDA_GRAPH`. `AUDIO8_TTS_ATTENTION_BACKEND` defaults to-`fa3`; set it to `flashinfer` for consumer Blackwell. Set+`AUDIO8_TTS_DISABLE_CUDA_GRAPH`. Torch compilation covers batch sizes through+4 by default when enabled; adjust `AUDIO8_TTS_TORCH_COMPILE_MAX_BS` for the+expected concurrency and available memory. `AUDIO8_TTS_ATTENTION_BACKEND`+defaults to `fa3`; set it to `flashinfer` for consumer Blackwell. Set`SGLANG_OMNI_SITE_PACKAGES` when the runtime dependencies are installed in aseparate site-packages directory.diff --git a/README_zh.md b/README_zh.mdindex 1f42c34..88f5d6b 100644--- a/README_zh.md+++ b/README_zh.md@@ -192,6 +192,8 @@ SGLANG_OMNI_PACKAGE="$(python3 -c 'import importlib.util, pathlib; s=importlib.uCUDA_VISIBLE_DEVICES=0 \SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \MODEL="${MODEL}" \+AUDIO8_TTS_ENABLE_TORCH_COMPILE=1 \+AUDIO8_TTS_TORCH_COMPILE_MAX_BS=4 \HOST=0.0.0.0 \PORT=8010 \./sglang_omni/scripts/run_server.sh@@ -212,9 +214,11 @@ MODEL="${MODEL}" \默认配置使用模型名 `audio8/tts-0.6b`、BF16、单卡、`0.2` 静态显存比例和最多 32 个并发请求。主要运行参数包括 `MODEL_NAME`、`AUDIO8_TTS_MEM_FRACTION_STATIC`、`AUDIO8_TTS_MAX_RUNNING_REQUESTS`、`AUDIO8_TTS_CHUNKED_PREFILL_SIZE` 和-`AUDIO8_TTS_DISABLE_CUDA_GRAPH`。`AUDIO8_TTS_ATTENTION_BACKEND` 默认为 `fa3`;消费级-Blackwell GPU 应设置为 `flashinfer`。如果运行依赖安装在单独的 site-packages 目录中,请-设置 `SGLANG_OMNI_SITE_PACKAGES`。+`AUDIO8_TTS_DISABLE_CUDA_GRAPH`。开启 Torch compile 后默认覆盖到 batch size 4,可根据+预期并发和可用显存通过 `AUDIO8_TTS_TORCH_COMPILE_MAX_BS` 调整上限。+`AUDIO8_TTS_ATTENTION_BACKEND` 默认为 `fa3`;消费级 Blackwell GPU 应设置为+`flashinfer`。如果运行依赖安装在单独的 site-packages 目录中,请设置+`SGLANG_OMNI_SITE_PACKAGES`。### 常见问题diff --git a/sglang_omni/OPTIMIZATION_REPORT.md b/sglang_omni/OPTIMIZATION_REPORT.mdindex e2d95bd..2b24e96 100644--- a/sglang_omni/OPTIMIZATION_REPORT.md+++ b/sglang_omni/OPTIMIZATION_REPORT.md@@ -36,13 +36,15 @@ Two environment variables expose SGLang's existing `torch.compile` support:```textAUDIO8_TTS_ENABLE_TORCH_COMPILE=0|1-AUDIO8_TTS_TORCH_COMPILE_MAX_BS=1+AUDIO8_TTS_TORCH_COMPILE_MAX_BS=4```-Compilation is disabled by default. For single-stream deployment, compile only-batch size 1 and let batch sizes 2 and above use the existing CUDA Graph path.-The first startup performs compilation and kernel autotuning; later startups-reuse `TORCHINDUCTOR_CACHE_DIR`.+Compilation is disabled by default. When enabled, the adapter compiles batch+sizes up to 4 by default, covering the CUDA Graph batch sizes validated here+(`1`, `2`, and `4`). Set `AUDIO8_TTS_TORCH_COMPILE_MAX_BS` to tune the limit for+the expected concurrency and available memory. Larger batches continue through+SGLang's existing dynamic batching path. The first startup performs compilation+and kernel autotuning; later startups reuse `TORCHINDUCTOR_CACHE_DIR`.### Optional static greedy path@@ -89,7 +91,7 @@ AUDIO8_TTS_MEM_FRACTION_STATIC=0.10 \AUDIO8_TTS_MAX_RUNNING_REQUESTS=4 \AUDIO8_TTS_ATTENTION_BACKEND=fa3 \AUDIO8_TTS_ENABLE_TORCH_COMPILE=1 \-AUDIO8_TTS_TORCH_COMPILE_MAX_BS=1 \+AUDIO8_TTS_TORCH_COMPILE_MAX_BS=4 \AUDIO8_TTS_GREEDY_FASTPATH=0 \sglang_omni/scripts/run_server.sh```diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.pyindex 24689a6..9eb3c97 100644--- a/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py@@ -129,7 +129,7 @@ def make_server_args(model_path: str) -> Any:os.getenv("AUDIO8_TTS_ENABLE_TORCH_COMPILE", "0") == "1")server_args.torch_compile_max_bs = int(- os.getenv("AUDIO8_TTS_TORCH_COMPILE_MAX_BS", "1")+ os.getenv("AUDIO8_TTS_TORCH_COMPILE_MAX_BS", "4"))attention_backend = os.getenv("AUDIO8_TTS_ATTENTION_BACKEND")if attention_backend: