Skip to content

SOTA-Class TTS at Compact Scale

Python

Enable Torch compile for batched inference

linux295650-uxc6e10eb2849f
diff --git a/README.md b/README.md
index 2bbc1ed..bcde98e 100644
--- a/README.md
+++ b/README.md
@@ -208,6 +208,8 @@ SGLANG_OMNI_PACKAGE="$(python3 -c 'import importlib.util, pathlib; s=importlib.u
CUDA_VISIBLE_DEVICES=0 \
SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \
MODEL="${MODEL}" \
+AUDIO8_TTS_ENABLE_TORCH_COMPILE=1 \
+AUDIO8_TTS_TORCH_COMPILE_MAX_BS=4 \
HOST=0.0.0.0 \
PORT=8010 \
./sglang_omni/scripts/run_server.sh
@@ -230,8 +232,10 @@ The defaults use model name `audio8/tts-0.6b`, BF16, one GPU, a `0.2` static
memory fraction, and up to 32 running requests. The main runtime controls are
`MODEL_NAME`, `AUDIO8_TTS_MEM_FRACTION_STATIC`,
`AUDIO8_TTS_MAX_RUNNING_REQUESTS`, `AUDIO8_TTS_CHUNKED_PREFILL_SIZE`, and
-`AUDIO8_TTS_DISABLE_CUDA_GRAPH`. `AUDIO8_TTS_ATTENTION_BACKEND` defaults to
-`fa3`; set it to `flashinfer` for consumer Blackwell. Set
+`AUDIO8_TTS_DISABLE_CUDA_GRAPH`. Torch compilation covers batch sizes through
+4 by default when enabled; adjust `AUDIO8_TTS_TORCH_COMPILE_MAX_BS` for the
+expected concurrency and available memory. `AUDIO8_TTS_ATTENTION_BACKEND`
+defaults to `fa3`; set it to `flashinfer` for consumer Blackwell. Set
`SGLANG_OMNI_SITE_PACKAGES` when the runtime dependencies are installed in a
separate site-packages directory.
diff --git a/README_zh.md b/README_zh.md
index 1f42c34..88f5d6b 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -192,6 +192,8 @@ SGLANG_OMNI_PACKAGE="$(python3 -c 'import importlib.util, pathlib; s=importlib.u
CUDA_VISIBLE_DEVICES=0 \
SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \
MODEL="${MODEL}" \
+AUDIO8_TTS_ENABLE_TORCH_COMPILE=1 \
+AUDIO8_TTS_TORCH_COMPILE_MAX_BS=4 \
HOST=0.0.0.0 \
PORT=8010 \
./sglang_omni/scripts/run_server.sh
@@ -212,9 +214,11 @@ MODEL="${MODEL}" \
默认配置使用模型名 `audio8/tts-0.6b`、BF16、单卡、`0.2` 静态显存比例和最多 32 个并发
请求。主要运行参数包括 `MODEL_NAME`、`AUDIO8_TTS_MEM_FRACTION_STATIC`、
`AUDIO8_TTS_MAX_RUNNING_REQUESTS`、`AUDIO8_TTS_CHUNKED_PREFILL_SIZE` 和
-`AUDIO8_TTS_DISABLE_CUDA_GRAPH`。`AUDIO8_TTS_ATTENTION_BACKEND` 默认为 `fa3`;消费级
-Blackwell GPU 应设置为 `flashinfer`。如果运行依赖安装在单独的 site-packages 目录中,请
-设置 `SGLANG_OMNI_SITE_PACKAGES`。
+`AUDIO8_TTS_DISABLE_CUDA_GRAPH`。开启 Torch compile 后默认覆盖到 batch size 4,可根据
+预期并发和可用显存通过 `AUDIO8_TTS_TORCH_COMPILE_MAX_BS` 调整上限。
+`AUDIO8_TTS_ATTENTION_BACKEND` 默认为 `fa3`;消费级 Blackwell GPU 应设置为
+`flashinfer`。如果运行依赖安装在单独的 site-packages 目录中,请设置
+`SGLANG_OMNI_SITE_PACKAGES`。
### 常见问题
diff --git a/sglang_omni/OPTIMIZATION_REPORT.md b/sglang_omni/OPTIMIZATION_REPORT.md
index e2d95bd..2b24e96 100644
--- a/sglang_omni/OPTIMIZATION_REPORT.md
+++ b/sglang_omni/OPTIMIZATION_REPORT.md
@@ -36,13 +36,15 @@ Two environment variables expose SGLang's existing `torch.compile` support:
```text
AUDIO8_TTS_ENABLE_TORCH_COMPILE=0|1
-AUDIO8_TTS_TORCH_COMPILE_MAX_BS=1
+AUDIO8_TTS_TORCH_COMPILE_MAX_BS=4
```
-Compilation is disabled by default. For single-stream deployment, compile only
-batch size 1 and let batch sizes 2 and above use the existing CUDA Graph path.
-The first startup performs compilation and kernel autotuning; later startups
-reuse `TORCHINDUCTOR_CACHE_DIR`.
+Compilation is disabled by default. When enabled, the adapter compiles batch
+sizes up to 4 by default, covering the CUDA Graph batch sizes validated here
+(`1`, `2`, and `4`). Set `AUDIO8_TTS_TORCH_COMPILE_MAX_BS` to tune the limit for
+the expected concurrency and available memory. Larger batches continue through
+SGLang's existing dynamic batching path. The first startup performs compilation
+and kernel autotuning; later startups reuse `TORCHINDUCTOR_CACHE_DIR`.
### Optional static greedy path
@@ -89,7 +91,7 @@ AUDIO8_TTS_MEM_FRACTION_STATIC=0.10 \
AUDIO8_TTS_MAX_RUNNING_REQUESTS=4 \
AUDIO8_TTS_ATTENTION_BACKEND=fa3 \
AUDIO8_TTS_ENABLE_TORCH_COMPILE=1 \
-AUDIO8_TTS_TORCH_COMPILE_MAX_BS=1 \
+AUDIO8_TTS_TORCH_COMPILE_MAX_BS=4 \
AUDIO8_TTS_GREEDY_FASTPATH=0 \
sglang_omni/scripts/run_server.sh
```
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py
index 24689a6..9eb3c97 100644
--- a/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py
@@ -129,7 +129,7 @@ def make_server_args(model_path: str) -> Any:
os.getenv("AUDIO8_TTS_ENABLE_TORCH_COMPILE", "0") == "1"
)
server_args.torch_compile_max_bs = int(
- os.getenv("AUDIO8_TTS_TORCH_COMPILE_MAX_BS", "1")
+ os.getenv("AUDIO8_TTS_TORCH_COMPILE_MAX_BS", "4")
)
attention_backend = os.getenv("AUDIO8_TTS_ATTENTION_BACKEND")
if attention_backend: