Skip to content

SOTA-Class TTS at Compact Scale

Python

Add SGLang Omni serving support

linyubupa2b2946e7f046
diff --git a/README.md b/README.md
index dc91ee7..20f5b19 100644
--- a/README.md
+++ b/README.md
@@ -122,6 +122,127 @@ The batch command writes `manifest.jsonl` and `failures.jsonl`. Existing WAV
files are skipped unless `--overwrite` is passed. See
`python audio8_tts_infer.py --help` for sampling and code-saving options.
+## SGLang Omni Serving
+
+The adapter in [`sglang_omni/`](sglang_omni/) provides an OpenAI-compatible
+service with SGLang paged attention, dynamic batching, a fixed KV cache for the
+fast codebook decoder, reference-audio encoding, and waveform decoding. It is
+installed as an independent `audio8_tts` model plugin and does not overwrite
+SGLang Omni core files.
+
+### Compatibility
+
+The adapter uses internal SGLang Omni interfaces, so deploy it with the tested
+revision instead of the latest `main` branch.
+
+| Dependency | Tested version |
+|---|---|
+| SGLang Omni | `68a572348837f7b004857b4b07993c20ade4c017` (`0.1.0`) |
+| SGLang | `0.5.8` |
+| PyTorch | `2.9.1+cu128` |
+| Transformers | `4.57.1` |
+| Precision | BF16 |
+
+### Install
+
+Run these commands from the Audio8 TTS repository root. The example uses
+Python 3.12 and [`uv`](https://docs.astral.sh/uv/).
+
+```bash
+export SGLANG_OMNI_ROOT=/opt/sglang-omni
+export MODEL=/models/Audio8-TTS-Preview-0.6b
+
+git clone https://github.com/sgl-project/sglang-omni.git "${SGLANG_OMNI_ROOT}"
+git -C "${SGLANG_OMNI_ROOT}" checkout 68a572348837f7b004857b4b07993c20ade4c017
+
+uv venv .venv-sglang --python 3.12
+source .venv-sglang/bin/activate
+uv pip install -v -e "${SGLANG_OMNI_ROOT}"
+
+hf download AutoArk-AI/Audio8-TTS-Preview-0.6b --local-dir "${MODEL}"
+./sglang_omni/scripts/install_adapter.sh "${SGLANG_OMNI_ROOT}"
+python3 ./sglang_omni/scripts/verify_install.py --model-path "${MODEL}"
+```
+
+For an existing wheel or site-packages installation, resolve the package
+directory and install the adapter there:
+
+```bash
+SGLANG_OMNI_PACKAGE="$(python3 -c 'import importlib.util, pathlib; s=importlib.util.find_spec("sglang_omni"); assert s and s.origin; print(pathlib.Path(s.origin).parent)')"
+./sglang_omni/scripts/install_adapter.sh "${SGLANG_OMNI_PACKAGE}"
+```
+
+### Start the service
+
+```bash
+CUDA_VISIBLE_DEVICES=0 \
+SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \
+MODEL="${MODEL}" \
+HOST=0.0.0.0 \
+PORT=8010 \
+./sglang_omni/scripts/run_server.sh
+```
+
+The defaults use model name `audio8/tts-0.6b`, BF16, one GPU, a `0.2` static
+memory fraction, and up to 32 running requests. The main runtime controls are
+`MODEL_NAME`, `AUDIO8_TTS_MEM_FRACTION_STATIC`,
+`AUDIO8_TTS_MAX_RUNNING_REQUESTS`, `AUDIO8_TTS_CHUNKED_PREFILL_SIZE`, and
+`AUDIO8_TTS_DISABLE_CUDA_GRAPH`. Set `SGLANG_OMNI_SITE_PACKAGES` when the
+runtime dependencies are installed in a separate site-packages directory.
+
+### Call the API
+
+Generate speech without a reference:
+
+```bash
+curl -sS --fail-with-body \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "model": "audio8/tts-0.6b",
+ "input": "Hello from Audio8 TTS.",
+ "response_format": "wav",
+ "max_new_tokens": 256,
+ "temperature": 0.8,
+ "top_p": 0.95,
+ "top_k": 50
+ }' \
+ http://127.0.0.1:8010/v1/audio/speech \
+ -o audio8.wav
+```
+
+Generate speech with one reference voice:
+
+```bash
+curl -sS --fail-with-body \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "model": "audio8/tts-0.6b",
+ "input": "This sentence uses the reference voice.",
+ "response_format": "wav",
+ "temperature": 0.8,
+ "top_p": 0.95,
+ "top_k": 50,
+ "references": [{
+ "audio_path": "/data/reference.wav",
+ "text": "The exact transcript of the reference recording."
+ }]
+ }' \
+ http://127.0.0.1:8010/v1/audio/speech \
+ -o audio8_clone.wav
+```
+
+The reference path must be visible inside the service environment. The current
+adapter supports TP=1, one reference per request, and non-streaming WAV output.
+Run the smoke test to verify a deployment:
+
+```bash
+BASE_URL=http://127.0.0.1:8010 ./sglang_omni/scripts/smoke_test.sh
+```
+
+To build the adapter into an existing image, append
+[`sglang_omni/Dockerfile.snippet`](sglang_omni/Dockerfile.snippet) after the
+SGLang Omni package and its Python dependencies are installed.
+
## Supervised Fine-tuning
Install the training dependencies first:
diff --git a/README_zh.md b/README_zh.md
index 2b1cf6d..4847dc8 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -112,6 +112,121 @@ python audio8_tts_infer.py \
批量命令会生成 `manifest.jsonl` 和 `failures.jsonl`。除非传入 `--overwrite`,否则已有 WAV 会
被跳过。采样和 codec codes 保存参数可通过 `python audio8_tts_infer.py --help` 查看。
+## SGLang Omni 服务部署
+
+[`sglang_omni/`](sglang_omni/) 中的适配器提供兼容 OpenAI API 的推理服务,包含 SGLang
+paged attention、动态 batching、Fast AR 固定 KV cache、参考音频编码和 waveform 解码。
+它以独立的 `audio8_tts` 模型插件安装,不会覆盖 SGLang Omni 核心文件。
+
+### 兼容版本
+
+适配器使用了 SGLang Omni 的内部接口,因此部署时应固定到以下已验证版本,不要直接使用最新
+`main` 分支。
+
+| 依赖 | 已验证版本 |
+|---|---|
+| SGLang Omni | `68a572348837f7b004857b4b07993c20ade4c017`(`0.1.0`) |
+| SGLang | `0.5.8` |
+| PyTorch | `2.9.1+cu128` |
+| Transformers | `4.57.1` |
+| 精度 | BF16 |
+
+### 安装
+
+在 Audio8 TTS 仓库根目录执行以下命令。示例使用 Python 3.12 和
+[`uv`](https://docs.astral.sh/uv/)。
+
+```bash
+export SGLANG_OMNI_ROOT=/opt/sglang-omni
+export MODEL=/models/Audio8-TTS-Preview-0.6b
+
+git clone https://github.com/sgl-project/sglang-omni.git "${SGLANG_OMNI_ROOT}"
+git -C "${SGLANG_OMNI_ROOT}" checkout 68a572348837f7b004857b4b07993c20ade4c017
+
+uv venv .venv-sglang --python 3.12
+source .venv-sglang/bin/activate
+uv pip install -v -e "${SGLANG_OMNI_ROOT}"
+
+hf download AutoArk-AI/Audio8-TTS-Preview-0.6b --local-dir "${MODEL}"
+./sglang_omni/scripts/install_adapter.sh "${SGLANG_OMNI_ROOT}"
+python3 ./sglang_omni/scripts/verify_install.py --model-path "${MODEL}"
+```
+
+如果 SGLang Omni 已通过 wheel 或 site-packages 安装,可以定位其包目录后安装适配器:
+
+```bash
+SGLANG_OMNI_PACKAGE="$(python3 -c 'import importlib.util, pathlib; s=importlib.util.find_spec("sglang_omni"); assert s and s.origin; print(pathlib.Path(s.origin).parent)')"
+./sglang_omni/scripts/install_adapter.sh "${SGLANG_OMNI_PACKAGE}"
+```
+
+### 启动服务
+
+```bash
+CUDA_VISIBLE_DEVICES=0 \
+SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \
+MODEL="${MODEL}" \
+HOST=0.0.0.0 \
+PORT=8010 \
+./sglang_omni/scripts/run_server.sh
+```
+
+默认配置使用模型名 `audio8/tts-0.6b`、BF16、单卡、`0.2` 静态显存比例和最多 32 个并发
+请求。主要运行参数包括 `MODEL_NAME`、`AUDIO8_TTS_MEM_FRACTION_STATIC`、
+`AUDIO8_TTS_MAX_RUNNING_REQUESTS`、`AUDIO8_TTS_CHUNKED_PREFILL_SIZE` 和
+`AUDIO8_TTS_DISABLE_CUDA_GRAPH`。如果运行依赖安装在单独的 site-packages 目录中,请设置
+`SGLANG_OMNI_SITE_PACKAGES`。
+
+### 调用 API
+
+无参考音频生成:
+
+```bash
+curl -sS --fail-with-body \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "model": "audio8/tts-0.6b",
+ "input": "你好,这是 Audio8 TTS 服务测试。",
+ "response_format": "wav",
+ "max_new_tokens": 256,
+ "temperature": 0.8,
+ "top_p": 0.95,
+ "top_k": 50
+ }' \
+ http://127.0.0.1:8010/v1/audio/speech \
+ -o audio8.wav
+```
+
+使用一个参考音色生成:
+
+```bash
+curl -sS --fail-with-body \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "model": "audio8/tts-0.6b",
+ "input": "这句话使用参考音色生成。",
+ "response_format": "wav",
+ "temperature": 0.8,
+ "top_p": 0.95,
+ "top_k": 50,
+ "references": [{
+ "audio_path": "/data/reference.wav",
+ "text": "参考录音对应的准确文本。"
+ }]
+ }' \
+ http://127.0.0.1:8010/v1/audio/speech \
+ -o audio8_clone.wav
+```
+
+参考音频路径必须在服务运行环境中可见。当前适配器支持 TP=1、每个请求一个 reference 和
+非流式 WAV 输出。运行以下 smoke test 可以验证部署:
+
+```bash
+BASE_URL=http://127.0.0.1:8010 ./sglang_omni/scripts/smoke_test.sh
+```
+
+构建镜像时,请在安装 SGLang Omni 包及其 Python 依赖后追加
+[`sglang_omni/Dockerfile.snippet`](sglang_omni/Dockerfile.snippet)。
+
## SFT 训练
先安装训练依赖:
diff --git a/sglang_omni/Dockerfile.snippet b/sglang_omni/Dockerfile.snippet
new file mode 100644
index 0000000..5ffd879
--- /dev/null
+++ b/sglang_omni/Dockerfile.snippet
@@ -0,0 +1,22 @@
+# SPDX-License-Identifier: Apache-2.0
+# Add these instructions to the SGLang-Omni image after its Python environment
+# and sglang_omni package have been installed.
+
+COPY sglang_omni /opt/audio8-adapter
+
+# This works for both wheel and editable installs. It resolves the package
+# without importing the full SGLang runtime during docker build.
+RUN SGLANG_OMNI_PACKAGE="$(python3 -c 'import importlib.util, pathlib; s=importlib.util.find_spec("sglang_omni"); assert s and s.origin; print(pathlib.Path(s.origin).parent)')" && \
+ /opt/audio8-adapter/scripts/install_adapter.sh "${SGLANG_OMNI_PACKAGE}" && \
+ python3 /opt/audio8-adapter/scripts/verify_install.py
+
+ENV AUDIO8_TTS_MEM_FRACTION_STATIC=0.2 \
+ AUDIO8_TTS_MAX_RUNNING_REQUESTS=32 \
+ AUDIO8_TTS_DISABLE_CUDA_GRAPH=0 \
+ FLASHINFER_WORKSPACE_BASE=/tmp/audio8-flashinfer
+
+# Mount or COPY the checkpoint to this path at runtime:
+# /models/Audio8-TTS-Preview-0.6b
+#
+# Example container command:
+# /opt/audio8-adapter/scripts/run_server.sh
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/__init__.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/__init__.py
new file mode 100644
index 0000000..56bc896
--- /dev/null
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/__init__.py
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+"""Audio8 TTS support for SGLang-Omni."""
+
+from . import config
+
+__all__ = ["config"]
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/config.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/config.py
new file mode 100644
index 0000000..5078e63
--- /dev/null
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/config.py
@@ -0,0 +1,47 @@
+# SPDX-License-Identifier: Apache-2.0
+from __future__ import annotations
+
+from typing import ClassVar
+
+from sglang_omni.config import ExecutorConfig, PipelineConfig, RelayConfig, StageConfig
+
+_PKG = "sglang_omni.models.audio8_tts.pipeline"
+
+
+class Audio8TTSPipelineConfig(PipelineConfig):
+ architecture: ClassVar[str] = "ArkttsModel"
+
+ model_path: str
+ entry_stage: str = "preprocessing"
+ stages: list[StageConfig] = [
+ StageConfig(
+ name="preprocessing",
+ executor=ExecutorConfig(
+ factory=f"{_PKG}.stages.create_preprocessing_executor",
+ args={"device": "cuda:0"},
+ ),
+ get_next=f"{_PKG}.next_stage.preprocessing_next",
+ relay=RelayConfig(device="cpu"),
+ ),
+ StageConfig(
+ name="tts_engine",
+ executor=ExecutorConfig(
+ factory=f"{_PKG}.stages.create_sglang_tts_engine_executor",
+ args={"device": "cuda:0", "max_new_tokens": 1024},
+ ),
+ get_next=f"{_PKG}.next_stage.tts_engine_next",
+ relay=RelayConfig(device="cuda"),
+ ),
+ StageConfig(
+ name="vocoder",
+ executor=ExecutorConfig(
+ factory=f"{_PKG}.stages.create_vocoder_executor",
+ args={"device": "cuda:0"},
+ ),
+ get_next=f"{_PKG}.next_stage.vocoder_next",
+ relay=RelayConfig(device="cpu"),
+ ),
+ ]
+
+
+EntryClass = Audio8TTSPipelineConfig
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py
new file mode 100644
index 0000000..ef04805
--- /dev/null
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py
@@ -0,0 +1,120 @@
+# SPDX-License-Identifier: Apache-2.0
+from __future__ import annotations
+
+import os
+from typing import Any
+
+from sglang_omni.engines.omni.engine import OmniEngine
+from sglang_omni.engines.omni.scheduler import Scheduler
+from sglang_omni.models.audio8_tts.runtime.audio8_sglang_ar import (
+ Audio8IterationController,
+ Audio8ModelRunner,
+ Audio8ResourceManager,
+)
+
+
+def create_audio8_engine(
+ server_args: Any,
+ *,
+ gpu_id: int,
+ eos_token_id: int,
+ max_new_tokens: int,
+) -> OmniEngine:
+ from sglang.srt.models.registry import ModelRegistry
+
+ from sglang_omni.engines.ar.sglang_backend.model_worker import (
+ ModelWorker,
+ ModelWorkerConfig,
+ )
+ from sglang_omni.engines.ar.sglang_backend.scheduler.cache import create_tree_cache
+ from sglang_omni.engines.ar.sglang_backend.scheduler.decode import DecodeManager
+ from sglang_omni.engines.ar.sglang_backend.scheduler.prefill import PrefillManager
+ from sglang_omni.engines.omni.runtime.sglang_ar import SGLangBatchPlanner
+ from sglang_omni.models.audio8_tts.sglang_model import Audio8SGLangModel
+
+ # Register lazily in the worker process so the adapter does not require a
+ # patch to SGLang-Omni's hard-coded model registry bootstrap.
+ ModelRegistry.models["ArkttsModel"] = Audio8SGLangModel
+
+ if server_args.attention_backend is None:
+ server_args.attention_backend = "fa3"
+ disable_attr = "disable_" + "cuda_graph"
+ want_cuda_graph = not bool(getattr(server_args, disable_attr, False))
+ setattr(server_args, disable_attr, True)
+ model_worker = ModelWorker(
+ config=ModelWorkerConfig(),
+ server_args=server_args,
+ gpu_id=gpu_id,
+ )
+ model_worker.model_runner.model.setup_audio8_decode(
+ server_args.max_running_requests
+ )
+ setattr(server_args, disable_attr, not want_cuda_graph)
+ if want_cuda_graph:
+ model_worker.model_runner.init_device_graphs()
+
+ request_pool, kv_allocator = model_worker.get_memory_pool()
+ tree_cache = create_tree_cache(
+ server_args,
+ request_pool,
+ kv_allocator,
+ server_args.page_size,
+ )
+ prefill_manager = PrefillManager(
+ page_size=server_args.page_size,
+ chunked_prefill_size=server_args.chunked_prefill_size,
+ max_prefill_tokens=server_args.max_prefill_tokens,
+ req_to_token_pool=request_pool,
+ token_to_kv_pool_allocator=kv_allocator,
+ tree_cache=tree_cache,
+ model_config=model_worker.model_config,
+ enable_overlap=False,
+ )
+ decode_manager = DecodeManager(
+ server_args=server_args,
+ token_to_kv_pool_allocator=kv_allocator,
+ on_retract=lambda request: prefill_manager.add_one_request(request),
+ )
+ planner = SGLangBatchPlanner(prefill_manager, decode_manager, server_args)
+ scheduler = Scheduler(
+ batch_planner=planner,
+ resource_manager=Audio8ResourceManager(
+ kv_allocator,
+ request_pool,
+ tree_cache,
+ ),
+ iteration_controller=Audio8IterationController(
+ tree_cache,
+ eos_token_id=eos_token_id,
+ max_new_tokens=max_new_tokens,
+ ),
+ stream_adapter=lambda request, output: (
+ None if output.data is None else output.data.codes
+ ),
+ )
+ return OmniEngine(
+ scheduler=scheduler,
+ model_runner=Audio8ModelRunner(model_worker, planner),
+ enable_overlap=False,
+ )
+
+
+def make_server_args(model_path: str) -> Any:
+ from sglang.srt.server_args import ServerArgs
+
+ server_args = ServerArgs(
+ model_path=model_path,
+ tp_size=1,
+ dtype="bfloat16",
+ trust_remote_code=True,
+ mem_fraction_static=float(os.getenv("AUDIO8_TTS_MEM_FRACTION_STATIC", "0.2")),
+ chunked_prefill_size=int(os.getenv("AUDIO8_TTS_CHUNKED_PREFILL_SIZE", "2048")),
+ max_running_requests=int(os.getenv("AUDIO8_TTS_MAX_RUNNING_REQUESTS", "32")),
+ disable_radix_cache=os.getenv("AUDIO8_TTS_DISABLE_RADIX_CACHE", "1") == "1",
+ )
+ setattr(
+ server_args,
+ "disable_" + "cuda_graph",
+ os.getenv("AUDIO8_TTS_DISABLE_CUDA_GRAPH", "0") == "1",
+ )
+ return server_args
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/io.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/io.py
new file mode 100644
index 0000000..0b3a5e6
--- /dev/null
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/io.py
@@ -0,0 +1,82 @@
+# SPDX-License-Identifier: Apache-2.0
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+import torch
+
+
+@dataclass
+class Audio8TTSState:
+ input_ids: Any = None
+ vq_mask_tokens: Any | None = None
+ vq_parts: Any | None = None
+ num_codebooks: int = 10
+ codebook_size: int = 4096
+ max_new_tokens: int = 1024
+ temperature: float = 0.8
+ top_p: float = 0.95
+ top_k: int = 50
+ do_sample: bool = True
+ output_codes: Any | None = None
+ prompt_tokens: int = 0
+ completion_tokens: int = 0
+ audio_samples: Any | None = None
+ sample_rate: int = 44100
+
+ @staticmethod
+ def _to_list(value: Any) -> Any:
+ return value.tolist() if isinstance(value, torch.Tensor) else value
+
+ def to_dict(self) -> dict[str, Any]:
+ data = {
+ "input_ids": self._to_list(self.input_ids),
+ "vq_mask_tokens": self._to_list(self.vq_mask_tokens),
+ "vq_parts": (
+ [self._to_list(part) for part in self.vq_parts]
+ if self.vq_parts is not None
+ else None
+ ),
+ "num_codebooks": self.num_codebooks,
+ "codebook_size": self.codebook_size,
+ "max_new_tokens": self.max_new_tokens,
+ "temperature": self.temperature,
+ "top_p": self.top_p,
+ "top_k": self.top_k,
+ "do_sample": self.do_sample,
+ "output_codes": self._to_list(self.output_codes),
+ "prompt_tokens": self.prompt_tokens,
+ "completion_tokens": self.completion_tokens,
+ "audio_samples": self._to_list(self.audio_samples),
+ "sample_rate": self.sample_rate,
+ }
+ return {key: value for key, value in data.items() if value is not None}
+
+ @classmethod
+ def from_dict(cls, data: dict[str, Any]) -> "Audio8TTSState":
+ vq_parts = data.get("vq_parts")
+ if vq_parts is not None:
+ vq_parts = [torch.as_tensor(part, dtype=torch.long) for part in vq_parts]
+ output_codes = data.get("output_codes")
+ return cls(
+ input_ids=data.get("input_ids"),
+ vq_mask_tokens=data.get("vq_mask_tokens"),
+ vq_parts=vq_parts,
+ num_codebooks=int(data.get("num_codebooks", 10)),
+ codebook_size=int(data.get("codebook_size", 4096)),
+ max_new_tokens=int(data.get("max_new_tokens", 1024)),
+ temperature=float(data.get("temperature", 0.8)),
+ top_p=float(data.get("top_p", 0.95)),
+ top_k=int(data.get("top_k", 50)),
+ do_sample=bool(data.get("do_sample", True)),
+ output_codes=(
+ torch.as_tensor(output_codes, dtype=torch.long)
+ if output_codes is not None
+ else None
+ ),
+ prompt_tokens=int(data.get("prompt_tokens", 0)),
+ completion_tokens=int(data.get("completion_tokens", 0)),
+ audio_samples=data.get("audio_samples"),
+ sample_rate=int(data.get("sample_rate", 44100)),
+ )
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/__init__.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/__init__.py
new file mode 100644
index 0000000..501ad20
--- /dev/null
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/__init__.py
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: Apache-2.0
+"""Audio8 TTS pipeline stages."""
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/engine_io.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/engine_io.py
new file mode 100644
index 0000000..704c780
--- /dev/null
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/engine_io.py
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: Apache-2.0
+from __future__ import annotations
+
+from typing import Any
+
+import torch
+
+from sglang_omni.models.audio8_tts.io import Audio8TTSState
+from sglang_omni.models.audio8_tts.runtime.audio8_sglang_ar import (
+ Audio8SGLangRequestData,
+)
+
+
+def build_tts_request(
+ state: Audio8TTSState,
+ tokenizer: Any,
+ request_id: str,
+) -> Audio8SGLangRequestData:
+ from sglang.srt.managers.schedule_batch import Req
+ from sglang.srt.sampling.sampling_params import SamplingParams
+
+ input_ids = list(state.input_ids)
+ sampling_params = SamplingParams(
+ max_new_tokens=state.max_new_tokens,
+ temperature=state.temperature,
+ )
+ req = Req(
+ rid=request_id,
+ origin_input_text="",
+ origin_input_ids=input_ids,
+ sampling_params=sampling_params,
+ vocab_size=tokenizer.vocab_size,
+ )
+ return Audio8SGLangRequestData(
+ input_ids=torch.tensor(input_ids, dtype=torch.long),
+ req=req,
+ vq_mask_tokens=(
+ torch.as_tensor(state.vq_mask_tokens, dtype=torch.bool)
+ if state.vq_mask_tokens is not None
+ else None
+ ),
+ vq_parts=(
+ [torch.as_tensor(part, dtype=torch.long) for part in state.vq_parts]
+ if state.vq_parts is not None
+ else None
+ ),
+ num_codebooks=state.num_codebooks,
+ codebook_size=state.codebook_size,
+ max_new_tokens=state.max_new_tokens,
+ temperature=state.temperature,
+ top_p=state.top_p,
+ top_k=state.top_k,
+ do_sample=state.do_sample,
+ )
+
+
+def apply_tts_result(state: Audio8TTSState, result: Audio8SGLangRequestData) -> None:
+ if result.output_codes:
+ all_codes = torch.cat(result.output_codes, dim=1)
+ state.output_codes = all_codes[1:]
+ state.completion_tokens = int(all_codes.shape[1])
+ else:
+ state.output_codes = None
+ state.prompt_tokens = len(result.input_ids) if result.input_ids is not None else 0
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/next_stage.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/next_stage.py
new file mode 100644
index 0000000..3ce60c0
--- /dev/null
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/next_stage.py
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: Apache-2.0
+from typing import Any
+
+
+def preprocessing_next(request_id: str, output: Any) -> str:
+ del request_id, output
+ return "tts_engine"
+
+
+def tts_engine_next(request_id: str, output: Any) -> str:
+ del request_id, output
+ return "vocoder"
+
+
+def vocoder_next(request_id: str, output: Any) -> None:
+ del request_id, output
+ return None
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/stages.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/stages.py
new file mode 100644
index 0000000..c69c496
--- /dev/null
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/stages.py
@@ -0,0 +1,261 @@
+# SPDX-License-Identifier: Apache-2.0
+from __future__ import annotations
+
+import base64
+import importlib.util
+import json
+import logging
+import sys
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any
+from urllib.parse import unquote_to_bytes
+
+import torch
+
+from sglang_omni.executors import EngineExecutor, PreprocessingExecutor
+from sglang_omni.models.audio8_tts.factory import create_audio8_engine, make_server_args
+from sglang_omni.models.audio8_tts.io import Audio8TTSState
+from sglang_omni.models.audio8_tts.pipeline.engine_io import (
+ apply_tts_result,
+ build_tts_request,
+)
+from sglang_omni.models.audio8_tts.pipeline.state_io import load_state, store_state
+from sglang_omni.models.audio8_tts.tokenizer import Audio8TokenizerAdapter, Reference
+from sglang_omni.proto import StagePayload
+
+logger = logging.getLogger(__name__)
+
+
+def _load_config(model_path: str) -> SimpleNamespace:
+ with open(Path(model_path) / "config.json", encoding="utf-8") as handle:
+ return SimpleNamespace(**json.load(handle))
+
+
+def _load_codec(model_path: str, device: str) -> Any:
+ model_dir = Path(model_path)
+ module_path = model_dir / "modeling_arktts_codec.py"
+ spec = importlib.util.spec_from_file_location("_audio8_arktts_codec", module_path)
+ if spec is None or spec.loader is None:
+ raise ImportError(f"Cannot import Audio8 codec from {module_path}")
+ module = importlib.util.module_from_spec(spec)
+ sys.modules[spec.name] = module
+ spec.loader.exec_module(module)
+ codec = module.ArkttsCodec(_load_config(model_path))
+ state = torch.load(
+ model_dir / "codec.pth",
+ map_location="cpu",
+ weights_only=True,
+ mmap=True,
+ )
+ if "state_dict" in state:
+ state = state["state_dict"]
+ if any("generator." in key for key in state):
+ state = {
+ key.replace("generator.", ""): value
+ for key, value in state.items()
+ if "generator." in key
+ }
+ state = {
+ key: value
+ for key, value in state.items()
+ if not key.endswith(("freqs_cis", "causal_mask"))
+ }
+ codec.load_state_dict(state, strict=True, assign=True)
+ dtype = torch.float32 if device == "cpu" else torch.bfloat16
+ return codec.eval().to(device=device, dtype=dtype)
+
+
+def _load_audio(audio_path: str, sample_rate: int) -> torch.Tensor:
+ import httpx
+
+ from sglang_omni.preprocessing.audio import AudioMediaIO
+
+ media_io = AudioMediaIO(target_sr=sample_rate)
+ if audio_path.startswith("data:"):
+ header, payload = audio_path.split(",", 1)
+ raw = base64.b64decode(payload) if ";base64" in header else unquote_to_bytes(payload)
+ audio, _ = media_io.load_bytes(raw)
+ elif audio_path.startswith(("http://", "https://")):
+ response = httpx.get(audio_path, follow_redirects=True, timeout=30)
+ response.raise_for_status()
+ audio, _ = media_io.load_bytes(response.content)
+ else:
+ audio, _ = media_io.load_file(Path(audio_path))
+ value = torch.from_numpy(audio).float()
+ if value.ndim == 1:
+ value = value.unsqueeze(0)
+ elif value.ndim == 2 and value.shape[0] > 1:
+ value = value.mean(dim=0, keepdim=True)
+ return value
+
+
+def _encode_reference(codec: Any, audio_path: str, device: str) -> torch.Tensor:
+ audio = _load_audio(audio_path, codec.sample_rate).to(
+ device=device,
+ dtype=next(codec.parameters()).dtype,
+ )
+ lengths = torch.tensor([audio.shape[-1]], device=device, dtype=torch.long)
+ with torch.inference_mode():
+ codes, code_lengths = codec.encode(audio, lengths)
+ return codes[0, :, : int(code_lengths[0].item())].cpu()
+
+
+def _validate_codes(value: Any, num_codebooks: int, codebook_size: int) -> torch.Tensor:
+ codes = torch.as_tensor(value, dtype=torch.long, device="cpu")
+ if codes.ndim != 2 or codes.shape[0] != num_codebooks or codes.shape[1] == 0:
+ raise ValueError(
+ f"reference codes must have shape [{num_codebooks}, T>0], got {tuple(codes.shape)}"
+ )
+ if int(codes.min().item()) < 0 or int(codes.max().item()) >= codebook_size:
+ raise ValueError(f"reference codes must be in [0, {codebook_size - 1}]")
+ return codes
+
+
+def create_preprocessing_executor(
+ model_path: str,
+ *,
+ device: str = "cuda:0",
+) -> PreprocessingExecutor:
+ from transformers import PreTrainedTokenizerFast
+
+ config = _load_config(model_path)
+ tokenizer = PreTrainedTokenizerFast.from_pretrained(
+ model_path,
+ fix_mistral_regex=False,
+ )
+ adapter = Audio8TokenizerAdapter(
+ tokenizer,
+ semantic_begin_id=config.semantic_begin_id,
+ semantic_end_id=config.semantic_end_id,
+ eos_token_id=config.eos_token_id,
+ )
+ codec: Any | None = None
+
+ def get_codec() -> Any:
+ nonlocal codec
+ if codec is None:
+ logger.info("Loading Audio8 reference codec on %s", device)
+ codec = _load_codec(model_path, device)
+ return codec
+
+ def preprocess(payload: StagePayload) -> StagePayload:
+ inputs = payload.request.inputs or {}
+ params = payload.request.params or {}
+ if isinstance(inputs, str):
+ inputs = {"text": inputs}
+ references: list[Reference] | None = None
+ raw_references = inputs.get("references")
+ if raw_references:
+ references = []
+ for raw in raw_references:
+ codes = raw.get("vq_codes")
+ if codes is None:
+ audio_path = raw.get("audio_path")
+ if not audio_path:
+ raise ValueError("reference requires audio_path or vq_codes")
+ codes = _encode_reference(get_codec(), audio_path, device)
+ codes = _validate_codes(codes, config.num_codebooks, config.codebook_size)
+ references.append(Reference(text=raw.get("text", ""), vq_codes=codes))
+ prompt = adapter.build_prompt(
+ inputs.get("text", ""),
+ references,
+ num_codebooks=config.num_codebooks,
+ )
+ state = Audio8TTSState(
+ input_ids=prompt["input_ids"],
+ vq_mask_tokens=prompt["vq_mask_tokens"],
+ vq_parts=prompt["vq_parts"],
+ num_codebooks=config.num_codebooks,
+ codebook_size=config.codebook_size,
+ max_new_tokens=int(params.get("max_new_tokens", 1024)),
+ temperature=float(params.get("temperature", 0.8)),
+ top_p=float(params.get("top_p", 0.95)),
+ top_k=int(params.get("top_k", 50)),
+ do_sample=float(params.get("temperature", 0.8)) > 0,
+ sample_rate=config.codec_sample_rate,
+ )
+ return store_state(payload, state)
+
+ return PreprocessingExecutor(preprocess)
+
+
+def create_sglang_tts_engine_executor(
+ model_path: str,
+ *,
+ device: str = "cuda:0",
+ max_new_tokens: int = 1024,
+) -> EngineExecutor:
+ from transformers import PreTrainedTokenizerFast
+
+ config = _load_config(model_path)
+ tokenizer = PreTrainedTokenizerFast.from_pretrained(
+ model_path,
+ fix_mistral_regex=False,
+ )
+ server_args = make_server_args(model_path)
+ gpu_id = int(device.split(":")[-1]) if ":" in device else 0
+ engine = create_audio8_engine(
+ server_args,
+ gpu_id=gpu_id,
+ eos_token_id=config.eos_token_id,
+ max_new_tokens=max_new_tokens,
+ )
+
+ def request_builder(payload: StagePayload):
+ return build_tts_request(load_state(payload), tokenizer, payload.request_id)
+
+ def result_builder(payload: StagePayload, result: Any) -> StagePayload:
+ state = load_state(payload)
+ apply_tts_result(state, result)
+ return store_state(payload, state)
+
+ return EngineExecutor(
+ engine=engine,
+ request_builder=request_builder,
+ result_builder=result_builder,
+ )
+
+
+def create_vocoder_executor(
+ model_path: str,
+ *,
+ device: str = "cuda:0",
+) -> PreprocessingExecutor:
+ codec = _load_codec(model_path, device)
+ config = _load_config(model_path)
+ warmup_codes = torch.zeros(
+ 1,
+ config.num_codebooks,
+ 4,
+ device=device,
+ dtype=torch.long,
+ )
+ with torch.inference_mode():
+ codec.decode(warmup_codes)
+
+ def vocode(payload: StagePayload) -> StagePayload:
+ state = load_state(payload)
+ if state.output_codes is None:
+ raise ValueError("Audio8 generation produced no codec frames")
+ codes = state.output_codes.to(device=device, dtype=torch.long)
+ with torch.inference_mode():
+ audio = codec.decode(codes.unsqueeze(0))[0, 0].float().cpu()
+ state.audio_samples = audio
+ state.sample_rate = codec.sample_rate
+ payload = store_state(payload, state)
+ payload.data.update(
+ {
+ "audio_data": audio.tolist(),
+ "sample_rate": codec.sample_rate,
+ "modality": "audio",
+ "usage": {
+ "prompt_tokens": state.prompt_tokens,
+ "completion_tokens": state.completion_tokens,
+ "total_tokens": state.prompt_tokens + state.completion_tokens,
+ },
+ }
+ )
+ return payload
+
+ return PreprocessingExecutor(vocode)
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/state_io.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/state_io.py
new file mode 100644
index 0000000..3607bbd
--- /dev/null
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/state_io.py
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: Apache-2.0
+from sglang_omni.models.audio8_tts.io import Audio8TTSState
+from sglang_omni.proto import StagePayload
+
+
+def load_state(payload: StagePayload) -> Audio8TTSState:
+ return Audio8TTSState.from_dict(payload.data)
+
+
+def store_state(payload: StagePayload, state: Audio8TTSState) -> StagePayload:
+ payload.data = state.to_dict()
+ return payload
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/runtime/__init__.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/runtime/__init__.py
new file mode 100644
index 0000000..8b2f235
--- /dev/null
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/runtime/__init__.py
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: Apache-2.0
+from .audio8_sglang_ar import Audio8SGLangRequestData, Audio8StepOutput
+
+__all__ = ["Audio8SGLangRequestData", "Audio8StepOutput"]
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/runtime/audio8_sglang_ar.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/runtime/audio8_sglang_ar.py
new file mode 100644
index 0000000..a6e036d
--- /dev/null
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/runtime/audio8_sglang_ar.py
@@ -0,0 +1,226 @@
+# SPDX-License-Identifier: Apache-2.0
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import TYPE_CHECKING, Any
+
+import torch
+from sglang.srt.mem_cache.common import release_kv_cache
+
+from sglang_omni.engines.omni.runtime.sglang_ar import (
+ SGLangARRequestData,
+ SGLangBatchPlanner,
+ SGLangResourceManager,
+)
+from sglang_omni.engines.omni.types import (
+ ModelRunnerOutput,
+ RequestOutput,
+ SchedulerOutput,
+ SchedulerRequest,
+)
+
+if TYPE_CHECKING:
+ from sglang_omni.engines.ar.sglang_backend.model_worker import ModelWorker
+
+
+@dataclass
+class Audio8StepOutput:
+ codes: torch.Tensor
+
+
+@dataclass
+class Audio8SGLangRequestData(SGLangARRequestData):
+ vq_mask_tokens: torch.Tensor | None = None
+ vq_parts: list[torch.Tensor] | None = None
+ num_codebooks: int = 10
+ codebook_size: int = 4096
+ output_codes: list[torch.Tensor] = field(default_factory=list)
+ max_new_tokens: int | None = None
+ temperature: float = 0.8
+ top_p: float = 0.95
+ top_k: int = 50
+ do_sample: bool = True
+ _previous_semantic_tokens: list[int] = field(default_factory=list)
+ _last_codebook_values: torch.Tensor | None = None
+
+
+class Audio8IterationController:
+ def __init__(self, tree_cache: Any, eos_token_id: int, max_new_tokens: int) -> None:
+ self.tree_cache = tree_cache
+ self.eos_token_id = int(eos_token_id)
+ self.max_new_tokens = int(max_new_tokens)
+
+ def update_request(self, request: SchedulerRequest, output: RequestOutput) -> None:
+ data: Audio8SGLangRequestData = request.data
+ req = data.req
+ if req.is_chunked > 0:
+ output.data = None
+ req.is_chunked -= 1
+ return
+ codes = output.data.codes.clone()
+ semantic = int(codes[0, -1].item())
+ if semantic != self.eos_token_id:
+ # HF generation does not send the EOS step to the waveform codec.
+ if data.output_codes:
+ data._previous_semantic_tokens.append(semantic)
+ data.output_codes.append(codes)
+ data._last_codebook_values = codes[1:, 0].clone()
+ req.output_ids.append(semantic)
+ if not req.finished() and req.decode_batch_idx == 0:
+ self.tree_cache.cache_unfinished_req(req)
+
+ def is_finished(self, request: SchedulerRequest, output: RequestOutput) -> bool:
+ data: Audio8SGLangRequestData = request.data
+ if data.req.is_chunked > 0:
+ return False
+ semantic = int(output.data.codes[0, -1].item())
+ if semantic == self.eos_token_id:
+ return True
+ return len(data.output_codes) >= (data.max_new_tokens or self.max_new_tokens)
+
+
+class Audio8ModelRunner:
+ def __init__(self, model_worker: "ModelWorker", batch_planner: SGLangBatchPlanner):
+ self.model_worker = model_worker
+ self.batch_planner = batch_planner
+
+ def _inject_reference_embeds(
+ self,
+ model_worker_batch: Any,
+ scheduler_output: SchedulerOutput,
+ ) -> None:
+ device = model_worker_batch.input_ids.device
+ model = self.model_worker.model_runner.model
+ text_embeds = model.get_embed_tokens()(model_worker_batch.input_ids)
+ offset = 0
+ for scheduled in scheduler_output.requests:
+ data: Audio8SGLangRequestData = scheduled.data
+ request_length = int(data.req.extend_input_len)
+ prefix_length = len(data.req.prefix_indices)
+ if data.vq_mask_tokens is not None and data.vq_parts:
+ mask = data.vq_mask_tokens.to(device=device, dtype=torch.bool).flatten()
+ expected_length = prefix_length + request_length
+ if mask.numel() != expected_length:
+ raise ValueError(
+ f"Audio8 reference mask length {mask.numel()} != request length {expected_length}"
+ )
+ mask_slice = mask[prefix_length : prefix_length + request_length]
+ parts = [part.to(device=device, dtype=torch.long).T for part in data.vq_parts]
+ all_codes = torch.cat(parts, dim=0)
+ if all_codes.shape[0] != int(mask.sum().item()):
+ raise ValueError("Audio8 reference mask/code length mismatch")
+ before = int(mask[:prefix_length].sum().item())
+ count = int(mask_slice.sum().item())
+ if count:
+ codes = all_codes[before : before + count]
+ if codes.shape[1] != model.config.num_codebooks:
+ raise ValueError("Audio8 reference has the wrong number of codebooks")
+ if int(codes.min().item()) < 0 or int(codes.max().item()) >= model.config.codebook_size:
+ raise ValueError("Audio8 reference code is outside the embedding range")
+ embedded_codes = codes + model._codebook_offsets[None]
+ codebook_sum = model.codebook_embeddings(embedded_codes).sum(dim=1)
+ indices = mask_slice.nonzero(as_tuple=True)[0] + offset
+ text_embeds[indices] += codebook_sum.to(text_embeds.dtype)
+ offset += request_length
+ model_worker_batch.input_embeds = text_embeds
+
+ def _update_runtime_buffers(
+ self,
+ model_worker_batch: Any,
+ scheduler_output: SchedulerOutput,
+ *,
+ is_prefill: bool,
+ ) -> None:
+ model = self.model_worker.model_runner.model
+ batch = len(scheduler_output.requests)
+ model._previous_valid[:batch].zero_()
+ if not is_prefill:
+ input_ids = model_worker_batch.input_ids
+ semantic_mask = (input_ids >= model.config.semantic_begin_id) & (
+ input_ids <= model.config.semantic_end_id
+ )
+ model._vq_mask[:batch].copy_(semantic_mask)
+ else:
+ model._vq_mask[:batch].zero_()
+
+ for index, scheduled in enumerate(scheduler_output.requests):
+ data: Audio8SGLangRequestData = scheduled.data
+ model._temperature[index] = max(float(data.temperature), 1e-5)
+ model._top_p[index] = min(max(float(data.top_p), 1e-5), 1.0)
+ model._top_k[index] = min(
+ max(int(data.top_k), 1),
+ int(model.config.codebook_size),
+ )
+ model._do_sample[index] = bool(data.do_sample)
+ history = data._previous_semantic_tokens[-model.config.ras_window_size :]
+ if history:
+ length = len(history)
+ model._previous_semantic[index, -length:] = torch.tensor(
+ history,
+ device=model._previous_semantic.device,
+ dtype=torch.long,
+ )
+ model._previous_valid[index, -length:] = True
+ if (
+ not is_prefill
+ and data._last_codebook_values is not None
+ and bool(model._vq_mask[index].item())
+ ):
+ model._vq_codes[index].copy_(data._last_codebook_values)
+
+ def _build_outputs(self, scheduler_output: SchedulerOutput) -> dict[str, RequestOutput]:
+ model = self.model_worker.model_runner.model
+ outputs: dict[str, RequestOutput] = {}
+ for index, scheduled in enumerate(scheduler_output.requests):
+ data: Audio8SGLangRequestData = scheduled.data
+ outputs[scheduled.request_id] = RequestOutput(
+ request_id=scheduled.request_id,
+ data=(
+ None
+ if data.req.is_chunked > 0
+ else Audio8StepOutput(model._output_codes[index].unsqueeze(-1))
+ ),
+ finished=False,
+ )
+ return outputs
+
+ def execute(self, scheduler_output: SchedulerOutput) -> ModelRunnerOutput:
+ from sglang.srt.model_executor.forward_batch_info import ForwardBatch
+
+ schedule_batch = scheduler_output.batch_data
+ worker_batch = schedule_batch.get_model_worker_batch()
+ is_prefill = schedule_batch.forward_mode.is_extend()
+ self._update_runtime_buffers(
+ worker_batch,
+ scheduler_output,
+ is_prefill=is_prefill,
+ )
+ if is_prefill:
+ self._inject_reference_embeds(worker_batch, scheduler_output)
+ forward_batch = ForwardBatch.init_new(worker_batch, self.model_worker.model_runner)
+ batch_result = self.model_worker.forward_batch_generation(forward_batch)
+ if schedule_batch.is_prefill_only:
+ batch_result.next_token_ids = torch.zeros(
+ len(worker_batch.seq_lens),
+ dtype=torch.long,
+ device=worker_batch.input_ids.device,
+ )
+ self.batch_planner.record_last_batch(schedule_batch)
+ outputs = self._build_outputs(scheduler_output)
+ model = self.model_worker.model_runner.model
+ batch = len(scheduler_output.requests)
+ schedule_batch.output_ids = model._output_semantic_ids[:batch].clone()
+ request_ids = [request.request_id for request in scheduler_output.requests]
+ return ModelRunnerOutput(
+ outputs=outputs,
+ req_ids=request_ids,
+ req_id_to_index={request_id: index for index, request_id in enumerate(request_ids)},
+ )
+
+
+class Audio8ResourceManager(SGLangResourceManager):
+ def free(self, request: SchedulerRequest) -> None:
+ data: Audio8SGLangRequestData = request.data
+ release_kv_cache(data.req, self.tree_cache)
+ data._previous_semantic_tokens.clear()
+ data._last_codebook_values = None
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/sglang_model.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/sglang_model.py
new file mode 100644
index 0000000..986432f
--- /dev/null
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/sglang_model.py
@@ -0,0 +1,771 @@
+# SPDX-License-Identifier: Apache-2.0
+"""SGLang-native slow AR plus Audio8's fixed-length fast codebook head."""
+
+from __future__ import annotations
+
+import logging
+from typing import Any, Iterable, Optional, Tuple
+
+import torch
+import torch.nn.functional as F
+from sgl_kernel.flash_attn import flash_attn_with_kvcache
+from sglang.srt.layers.logits_processor import LogitsProcessorOutput
+from torch import Tensor, nn
+
+from sglang_omni.vendor.sglang.core import ForwardBatch
+from sglang_omni.vendor.sglang.layers import (
+ MergedColumnParallelLinear,
+ QKVParallelLinear,
+ RadixAttention,
+ RMSNorm,
+ RowParallelLinear,
+ VocabParallelEmbedding,
+ get_rope,
+)
+from sglang_omni.vendor.sglang.models import apply_qk_norm
+from sglang_omni.vendor.sglang.utils import make_layers
+
+logger = logging.getLogger(__name__)
+
+
+ "audio8::fast_attention_with_cache",
+ mutates_args=("key_cache", "value_cache"),
+)
+def _audio8_fast_attention_with_cache(
+ query: Tensor,
+ key_cache: Tensor,
+ value_cache: Tensor,
+ key: Tensor,
+ value: Tensor,
+ cache_positions: Tensor,
+) -> Tensor:
+ return flash_attn_with_kvcache(
+ q=query,
+ k_cache=key_cache,
+ v_cache=value_cache,
+ k=key,
+ v=value,
+ cache_seqlens=cache_positions.contiguous(),
+ causal=True,
+ num_splits=0,
+ )
+
+
+@_audio8_fast_attention_with_cache.register_fake
+def _audio8_fast_attention_with_cache_fake(
+ query: Tensor,
+ key_cache: Tensor,
+ value_cache: Tensor,
+ key: Tensor,
+ value: Tensor,
+ cache_positions: Tensor,
+) -> Tensor:
+ del key_cache, value_cache, key, value, cache_positions
+ return torch.empty_like(query)
+
+
+class Audio8FastKVCache(nn.Module):
+ def __init__(
+ self,
+ max_batch_size: int,
+ max_sequence_length: int,
+ num_kv_heads: int,
+ head_dim: int,
+ *,
+ device: torch.device,
+ dtype: torch.dtype,
+ ) -> None:
+ super().__init__()
+ shape = (
+ max_batch_size,
+ max_sequence_length,
+ num_kv_heads,
+ head_dim,
+ )
+ self.register_buffer(
+ "key",
+ torch.zeros(shape, device=device, dtype=dtype),
+ persistent=False,
+ )
+ self.register_buffer(
+ "value",
+ torch.zeros(shape, device=device, dtype=dtype),
+ persistent=False,
+ )
+
+ def for_batch(self, batch_size: int) -> tuple[Tensor, Tensor]:
+ return self.key[:batch_size], self.value[:batch_size]
+
+ def clear(self) -> None:
+ self.key.zero_()
+ self.value.zero_()
+
+
+class Audio8Attention(nn.Module):
+ def __init__(
+ self,
+ *,
+ hidden_size: int,
+ num_heads: int,
+ num_kv_heads: int,
+ head_dim: int,
+ layer_id: int,
+ rope_base: float,
+ max_position_embeddings: int,
+ rms_norm_eps: float,
+ qkv_bias: bool,
+ output_bias: bool,
+ qk_norm: bool,
+ ) -> None:
+ super().__init__()
+ self.q_size = num_heads * head_dim
+ self.kv_size = num_kv_heads * head_dim
+ self.head_dim = head_dim
+ self.qkv_proj = QKVParallelLinear(
+ hidden_size,
+ head_dim,
+ num_heads,
+ num_kv_heads,
+ bias=qkv_bias,
+ )
+ self.o_proj = RowParallelLinear(
+ num_heads * head_dim,
+ hidden_size,
+ bias=output_bias,
+ )
+ self.rotary_emb = get_rope(
+ head_dim,
+ rotary_dim=head_dim,
+ max_position=max_position_embeddings,
+ base=rope_base,
+ is_neox_style=False,
+ )
+ self.attn = RadixAttention(
+ num_heads,
+ head_dim,
+ head_dim**-0.5,
+ num_kv_heads=num_kv_heads,
+ layer_id=layer_id,
+ )
+ self.qk_norm = qk_norm
+ if qk_norm:
+ self.q_norm = RMSNorm(head_dim, eps=rms_norm_eps)
+ self.k_norm = RMSNorm(head_dim, eps=rms_norm_eps)
+
+ def forward(
+ self,
+ positions: Tensor,
+ hidden_states: Tensor,
+ forward_batch: ForwardBatch,
+ ) -> Tensor:
+ qkv, _ = self.qkv_proj(hidden_states)
+ q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
+ if self.qk_norm:
+ q, k = apply_qk_norm(q, k, self.q_norm, self.k_norm, self.head_dim)
+ q, k = self.rotary_emb(positions, q, k)
+ output = self.attn(q, k, v, forward_batch)
+ output, _ = self.o_proj(output)
+ return output
+
+
+class Audio8DecoderLayer(nn.Module):
+ def __init__(self, config: Any, layer_id: int) -> None:
+ super().__init__()
+ self.self_attn = Audio8Attention(
+ hidden_size=config.dim,
+ num_heads=config.n_head,
+ num_kv_heads=config.n_local_heads,
+ head_dim=config.head_dim,
+ layer_id=layer_id,
+ rope_base=config.rope_base,
+ max_position_embeddings=config.max_seq_len,
+ rms_norm_eps=config.norm_eps,
+ qkv_bias=config.attention_qkv_bias,
+ output_bias=config.attention_o_bias,
+ qk_norm=config.attention_qk_norm,
+ )
+ self.gate_up_proj = MergedColumnParallelLinear(
+ config.dim,
+ [config.intermediate_size, config.intermediate_size],
+ bias=False,
+ )
+ self.down_proj = RowParallelLinear(
+ config.intermediate_size,
+ config.dim,
+ bias=False,
+ )
+ self.input_layernorm = RMSNorm(config.dim, eps=config.norm_eps)
+ self.post_attention_layernorm = RMSNorm(config.dim, eps=config.norm_eps)
+
+ def forward(
+ self,
+ positions: Tensor,
+ hidden_states: Tensor,
+ forward_batch: ForwardBatch,
+ residual: Optional[Tensor],
+ ) -> Tuple[Tensor, Tensor]:
+ if residual is None:
+ residual = hidden_states
+ hidden_states = self.input_layernorm(hidden_states)
+ else:
+ hidden_states, residual = self.input_layernorm(hidden_states, residual)
+ hidden_states = self.self_attn(positions, hidden_states, forward_batch)
+ hidden_states, residual = self.post_attention_layernorm(hidden_states, residual)
+ gate_up, _ = self.gate_up_proj(hidden_states)
+ gate, up = gate_up.chunk(2, dim=-1)
+ hidden_states = F.silu(gate) * up
+ hidden_states, _ = self.down_proj(hidden_states)
+ return hidden_states, residual
+
+
+class FastRMSNorm(nn.Module):
+ def __init__(self, dim: int, eps: float) -> None:
+ super().__init__()
+ self.eps = float(eps)
+ self.weight = nn.Parameter(torch.ones(dim))
+
+ def forward(self, value: Tensor) -> Tensor:
+ normalized = value.float() * torch.rsqrt(
+ value.float().pow(2).mean(-1, keepdim=True) + self.eps
+ )
+ return normalized.to(value.dtype) * self.weight
+
+
+def _rope(length: int, head_dim: int, base: float, device: torch.device) -> Tensor:
+ frequencies = 1.0 / (
+ base ** (torch.arange(0, head_dim, 2, device=device).float() / head_dim)
+ )
+ phases = torch.outer(torch.arange(length, device=device), frequencies)
+ values = torch.polar(torch.ones_like(phases), phases)
+ return torch.stack((values.real, values.imag), dim=-1).to(torch.bfloat16)
+
+
+def _apply_rope(value: Tensor, rope_values: Tensor) -> Tensor:
+ shaped = value.float().reshape(*value.shape[:-1], -1, 2)
+ rope_values = rope_values[None, :, None]
+ output = torch.stack(
+ (
+ shaped[..., 0] * rope_values[..., 0]
+ - shaped[..., 1] * rope_values[..., 1],
+ shaped[..., 1] * rope_values[..., 0]
+ + shaped[..., 0] * rope_values[..., 1],
+ ),
+ dim=-1,
+ )
+ return output.flatten(3).to(value.dtype)
+
+
+class FastAttention(nn.Module):
+ def __init__(self, config: Any) -> None:
+ super().__init__()
+ total = (config.fast_n_head + 2 * config.fast_n_local_heads) * config.fast_head_dim
+ self.wqkv = nn.Linear(
+ config.fast_dim,
+ total,
+ bias=config.fast_attention_qkv_bias,
+ )
+ self.wo = nn.Linear(
+ config.fast_n_head * config.fast_head_dim,
+ config.fast_dim,
+ bias=config.fast_attention_o_bias,
+ )
+ self.n_head = config.fast_n_head
+ self.n_local_heads = config.fast_n_local_heads
+ self.head_dim = config.fast_head_dim
+ self.qk_norm = config.fast_attention_qk_norm
+ self.audio8_cache: Audio8FastKVCache | None = None
+ if self.qk_norm:
+ self.q_norm = FastRMSNorm(self.head_dim, config.norm_eps)
+ self.k_norm = FastRMSNorm(self.head_dim, config.norm_eps)
+
+ def setup_audio8_cache(
+ self,
+ max_batch_size: int,
+ max_sequence_length: int,
+ *,
+ device: torch.device,
+ dtype: torch.dtype,
+ ) -> None:
+ self.audio8_cache = Audio8FastKVCache(
+ max_batch_size,
+ max_sequence_length,
+ self.n_local_heads,
+ self.head_dim,
+ device=device,
+ dtype=dtype,
+ )
+
+ def clear_audio8_cache(self) -> None:
+ if self.audio8_cache is None:
+ raise RuntimeError("Audio8 fast KV cache has not been initialized")
+ self.audio8_cache.clear()
+
+ def forward(self, value: Tensor, rope_values: Tensor) -> Tensor:
+ batch, length, _ = value.shape
+ q_size = self.n_head * self.head_dim
+ kv_size = self.n_local_heads * self.head_dim
+ q, k, v = self.wqkv(value).split((q_size, kv_size, kv_size), dim=-1)
+ q = q.view(batch, length, self.n_head, self.head_dim)
+ k = k.view(batch, length, self.n_local_heads, self.head_dim)
+ v = v.view(batch, length, self.n_local_heads, self.head_dim)
+ if self.qk_norm:
+ q = self.q_norm(q)
+ k = self.k_norm(k)
+ q = _apply_rope(q, rope_values).transpose(1, 2)
+ k = _apply_rope(k, rope_values).transpose(1, 2)
+ v = v.transpose(1, 2)
+ repeat = self.n_head // self.n_local_heads
+ k = k.repeat_interleave(repeat, dim=1)
+ v = v.repeat_interleave(repeat, dim=1)
+ scores = q @ k.transpose(-2, -1) / (self.head_dim**0.5)
+ causal = torch.ones(
+ length,
+ length,
+ dtype=torch.bool,
+ device=value.device,
+ ).tril()
+ scores = scores.masked_fill(~causal[None, None], -float("inf"))
+ output = torch.softmax(scores, dim=-1) @ v
+ return self.wo(output.transpose(1, 2).contiguous().view(batch, length, q_size))
+
+ def forward_audio8_cached(
+ self,
+ value: Tensor,
+ rope_values: Tensor,
+ cache_positions: Tensor,
+ ) -> Tensor:
+ batch, length, _ = value.shape
+ if length != 1:
+ raise ValueError("Audio8 fast KV-cache decode expects one token per step")
+ q_size = self.n_head * self.head_dim
+ kv_size = self.n_local_heads * self.head_dim
+ q, k, v = self.wqkv(value).split((q_size, kv_size, kv_size), dim=-1)
+ q = q.view(batch, length, self.n_head, self.head_dim)
+ k = k.view(batch, length, self.n_local_heads, self.head_dim)
+ v = v.view(batch, length, self.n_local_heads, self.head_dim)
+ if self.qk_norm:
+ q = self.q_norm(q)
+ k = self.k_norm(k)
+ q = _apply_rope(q, rope_values)
+ k = _apply_rope(k, rope_values)
+ if self.audio8_cache is None:
+ raise RuntimeError("Audio8 fast KV cache has not been initialized")
+ key_cache, value_cache = self.audio8_cache.for_batch(batch)
+ output = _audio8_fast_attention_with_cache(
+ q,
+ key_cache,
+ value_cache,
+ k,
+ v,
+ cache_positions,
+ )
+ return self.wo(output.contiguous().view(batch, length, q_size))
+
+
+class FastFeedForward(nn.Module):
+ def __init__(self, config: Any) -> None:
+ super().__init__()
+ self.w1 = nn.Linear(config.fast_dim, config.fast_intermediate_size, bias=False)
+ self.w2 = nn.Linear(config.fast_intermediate_size, config.fast_dim, bias=False)
+ self.w3 = nn.Linear(config.fast_dim, config.fast_intermediate_size, bias=False)
+
+ def forward(self, value: Tensor) -> Tensor:
+ return self.w2(F.silu(self.w1(value)) * self.w3(value))
+
+
+class FastDecoderLayer(nn.Module):
+ def __init__(self, config: Any) -> None:
+ super().__init__()
+ self.attention = FastAttention(config)
+ self.feed_forward = FastFeedForward(config)
+ self.ffn_norm = FastRMSNorm(config.fast_dim, config.norm_eps)
+ self.attention_norm = FastRMSNorm(config.fast_dim, config.norm_eps)
+
+ def forward(self, value: Tensor, rope_values: Tensor) -> Tensor:
+ hidden = value + self.attention(self.attention_norm(value), rope_values)
+ return hidden + self.feed_forward(self.ffn_norm(hidden))
+
+ def forward_audio8_cached(
+ self,
+ value: Tensor,
+ rope_values: Tensor,
+ cache_positions: Tensor,
+ ) -> Tensor:
+ hidden = value + self.attention.forward_audio8_cached(
+ self.attention_norm(value),
+ rope_values,
+ cache_positions,
+ )
+ return hidden + self.feed_forward(self.ffn_norm(hidden))
+
+
+class Audio8SGLangModel(nn.Module):
+ def __init__(self, config: Any, quant_config: Any = None) -> None:
+ super().__init__()
+ del quant_config
+ self.config = config
+ self.vocab_size = int(config.vocab_size)
+ self.hidden_size = int(config.dim)
+ self.num_layers = int(config.n_layer)
+ self.tie_word_embeddings = bool(config.tie_word_embeddings)
+ self.embed_tokens = VocabParallelEmbedding(config.vocab_size, config.dim)
+ self.codebook_embeddings = nn.Embedding(
+ config.codebook_size * config.num_codebooks,
+ config.dim,
+ )
+ self.start_layer = 0
+ self.end_layer = config.n_layer
+ self.layers = make_layers(
+ config.n_layer,
+ lambda idx, prefix: Audio8DecoderLayer(config, idx),
+ prefix="layers",
+ )
+ self.norm = RMSNorm(config.dim, eps=config.norm_eps)
+
+ self.fast_project_in = (
+ nn.Linear(config.dim, config.fast_dim)
+ if config.fast_dim != config.dim
+ else nn.Identity()
+ )
+ self.fast_embeddings = nn.Embedding(config.codebook_size, config.fast_dim)
+ self.fast_layers = nn.ModuleList(
+ [FastDecoderLayer(config) for _ in range(config.n_fast_layer)]
+ )
+ self.fast_norm = FastRMSNorm(config.fast_dim, config.norm_eps)
+ self.fast_output = nn.Linear(config.fast_dim, config.codebook_size, bias=False)
+ self._decode_ready = False
+
+ def setup_audio8_decode(self, max_batch_size: int) -> None:
+ device = self.embed_tokens.weight.device
+ config = self.config
+ self.register_buffer(
+ "_codebook_offsets",
+ torch.arange(config.num_codebooks, device=device, dtype=torch.long)
+ * config.codebook_size,
+ persistent=False,
+ )
+ self.register_buffer(
+ "_vq_codes",
+ torch.zeros(max_batch_size, config.num_codebooks, device=device, dtype=torch.long),
+ persistent=False,
+ )
+ self.register_buffer(
+ "_vq_mask",
+ torch.zeros(max_batch_size, device=device, dtype=torch.bool),
+ persistent=False,
+ )
+ semantic_bias = torch.full(
+ (config.vocab_size,),
+ -float("inf"),
+ device=device,
+ dtype=torch.float32,
+ )
+ semantic_bias[config.semantic_begin_id : config.semantic_end_id + 1] = 0.0
+ semantic_bias[config.eos_token_id] = 0.0
+ self.register_buffer("_semantic_bias", semantic_bias, persistent=False)
+ self.register_buffer(
+ "_output_codes",
+ torch.zeros(
+ max_batch_size,
+ config.num_codebooks + 1,
+ device=device,
+ dtype=torch.long,
+ ),
+ persistent=False,
+ )
+ self.register_buffer(
+ "_output_semantic_ids",
+ torch.zeros(max_batch_size, device=device, dtype=torch.long),
+ persistent=False,
+ )
+ self.register_buffer(
+ "_temperature",
+ torch.full((max_batch_size,), 0.8, device=device),
+ persistent=False,
+ )
+ self.register_buffer(
+ "_top_p",
+ torch.full((max_batch_size,), 0.95, device=device),
+ persistent=False,
+ )
+ self.register_buffer(
+ "_top_k",
+ torch.full((max_batch_size,), 50, device=device, dtype=torch.long),
+ persistent=False,
+ )
+ self.register_buffer(
+ "_do_sample",
+ torch.ones(max_batch_size, device=device, dtype=torch.bool),
+ persistent=False,
+ )
+ self.register_buffer(
+ "_previous_semantic",
+ torch.zeros(
+ max_batch_size,
+ config.ras_window_size,
+ device=device,
+ dtype=torch.long,
+ ),
+ persistent=False,
+ )
+ self.register_buffer(
+ "_previous_valid",
+ torch.zeros(
+ max_batch_size,
+ config.ras_window_size,
+ device=device,
+ dtype=torch.bool,
+ ),
+ persistent=False,
+ )
+ self.register_buffer(
+ "_fast_rope",
+ _rope(
+ config.num_codebooks,
+ config.fast_head_dim,
+ config.rope_base,
+ device,
+ ),
+ persistent=False,
+ )
+ self.register_buffer(
+ "_audio8_fast_position",
+ torch.zeros(1, device=device, dtype=torch.long),
+ persistent=False,
+ )
+ for layer in self.fast_layers:
+ layer.attention.setup_audio8_cache(
+ max_batch_size,
+ config.num_codebooks + 1,
+ device=device,
+ dtype=self.embed_tokens.weight.dtype,
+ )
+ self._decode_ready = True
+
+ def _embed_decode(self, input_ids: Tensor) -> Tensor:
+ hidden = self.embed_tokens(input_ids)
+ batch = hidden.shape[0]
+ codes = self._vq_codes[:batch] + self._codebook_offsets[None]
+ codebook_sum = self.codebook_embeddings(codes).sum(dim=1).to(hidden.dtype)
+ return torch.where(
+ self._vq_mask[:batch, None],
+ hidden + codebook_sum,
+ hidden,
+ )
+
+ @staticmethod
+ def _sample(
+ scores: Tensor,
+ temperature: Tensor,
+ top_p: Tensor,
+ top_k: Tensor,
+ do_sample: Tensor,
+ ) -> Tensor:
+ sorted_scores, sorted_indices = torch.sort(scores, descending=True, dim=-1)
+ cumulative = torch.softmax(sorted_scores, dim=-1).cumsum(dim=-1)
+ positions = torch.arange(scores.shape[-1], device=scores.device)[None]
+ remove_sorted = (cumulative > top_p[:, None]) | (positions >= top_k[:, None])
+ remove_sorted[:, 0] = False
+ remove = torch.zeros_like(remove_sorted).scatter(1, sorted_indices, remove_sorted)
+ filtered = scores.masked_fill(remove, -float("inf"))
+ filtered = filtered / temperature[:, None].clamp_min(1e-5)
+ greedy = filtered.argmax(dim=-1)
+ probabilities = torch.softmax(filtered, dim=-1)
+ random = torch.rand_like(probabilities).clamp_min(torch.finfo(probabilities.dtype).tiny)
+ sampled = torch.argmax(probabilities / (-torch.log(random)), dim=-1)
+ return torch.where(do_sample, sampled, greedy)
+
+ def _sample_semantic(self, logits: Tensor) -> Tensor:
+ batch = logits.shape[0]
+ scores = logits.float() + self._semantic_bias
+ normal = self._sample(
+ scores,
+ self._temperature[:batch],
+ self._top_p[:batch],
+ self._top_k[:batch],
+ self._do_sample[:batch],
+ )
+ ras_temperature = torch.full_like(
+ self._temperature[:batch], self.config.ras_temperature
+ )
+ ras_top_p = torch.full_like(self._top_p[:batch], self.config.ras_top_p)
+ high = self._sample(
+ scores,
+ ras_temperature,
+ ras_top_p,
+ self._top_k[:batch],
+ self._do_sample[:batch],
+ )
+ repeated = (
+ (self._previous_semantic[:batch] == normal[:, None])
+ & self._previous_valid[:batch]
+ ).any(dim=1)
+ is_semantic = (normal >= self.config.semantic_begin_id) & (
+ normal <= self.config.semantic_end_id
+ )
+ return torch.where(repeated & is_semantic, high, normal)
+
+ def _clear_audio8_fast_caches(self) -> None:
+ for layer in self.fast_layers:
+ layer.attention.clear_audio8_cache()
+
+ def _run_audio8_fast_position(self, hidden: Tensor, position: int) -> Tensor:
+ batch = hidden.shape[0]
+ self._audio8_fast_position.fill_(position)
+ rope_values = self._fast_rope[self._audio8_fast_position]
+ cache_positions = self._audio8_fast_position.expand(batch).to(torch.int32)
+ for layer in self.fast_layers:
+ hidden = layer.forward_audio8_cached(
+ hidden,
+ rope_values,
+ cache_positions,
+ )
+ return self.fast_output(self.fast_norm(hidden))[:, -1]
+
+ @torch.no_grad()
+ def _decode_codebooks(self, logits: Tensor, slow_hidden: Tensor) -> None:
+ batch = logits.shape[0]
+ semantic = self._sample_semantic(logits)
+ current = (semantic - self.config.semantic_begin_id).clamp(
+ 0, self.config.codebook_size - 1
+ )
+ self._output_codes[:batch, 0] = semantic
+ self._output_codes[:batch, 1] = current
+ self._clear_audio8_fast_caches()
+ fast_hidden = self.fast_project_in(slow_hidden).unsqueeze(1)
+ self._run_audio8_fast_position(fast_hidden, 0)
+ for position in range(1, self.config.num_codebooks):
+ fast_hidden = self.fast_embeddings(current).unsqueeze(1)
+ scores = self._run_audio8_fast_position(fast_hidden, position).float()
+ current = self._sample(
+ scores,
+ self._temperature[:batch],
+ self._top_p[:batch],
+ self._top_k[:batch],
+ self._do_sample[:batch],
+ )
+ self._output_codes[:batch, position + 1] = current
+ self._output_semantic_ids[:batch] = semantic
+
+ def forward(
+ self,
+ input_ids: Tensor,
+ positions: Tensor,
+ forward_batch: ForwardBatch,
+ input_embeds: Optional[Tensor] = None,
+ ) -> LogitsProcessorOutput:
+ if input_embeds is None and forward_batch.input_embeds is not None:
+ input_embeds = forward_batch.input_embeds
+ if input_embeds is not None:
+ hidden_states = input_embeds
+ elif self._decode_ready:
+ hidden_states = self._embed_decode(input_ids)
+ else:
+ hidden_states = self.embed_tokens(input_ids)
+
+ residual = None
+ for layer_index in range(self.start_layer, self.end_layer):
+ hidden_states, residual = self.layers[layer_index](
+ positions, hidden_states, forward_batch, residual
+ )
+ hidden_states, _ = self.norm(hidden_states, residual)
+ if forward_batch.forward_mode.is_extend():
+ last_index = torch.cumsum(forward_batch.extend_seq_lens, dim=0) - 1
+ hidden_states = hidden_states[last_index]
+ logits = F.linear(hidden_states, self.embed_tokens.weight)
+ if self._decode_ready:
+ self._decode_codebooks(logits, hidden_states)
+ return LogitsProcessorOutput(
+ next_token_logits=logits,
+ hidden_states=hidden_states,
+ )
+
+ def get_embed_tokens(self) -> nn.Module:
+ return self.embed_tokens
+
+ def load_weights(self, weights: Iterable[Tuple[str, Tensor]]) -> None:
+ params = dict(self.named_parameters())
+ for name, loaded_weight in weights:
+ if name in {"freqs_cis", "fast_freqs_cis"}:
+ continue
+ if self._load_slow_weight(name, loaded_weight, params):
+ continue
+ target_name = "embed_tokens.weight" if name == "embeddings.weight" else name
+ target = params.get(target_name)
+ if target is None:
+ logger.debug("Skipping Audio8 weight: %s", name)
+ continue
+ loader = getattr(target, "weight_loader", _default_weight_loader)
+ loader(target, loaded_weight)
+
+ def _load_slow_weight(
+ self,
+ name: str,
+ loaded_weight: Tensor,
+ params: dict[str, nn.Parameter],
+ ) -> bool:
+ if not name.startswith("layers."):
+ return False
+ remap: dict[str, Any] = {
+ "attention.wqkv.weight": None,
+ "attention.wqkv.bias": None,
+ "attention.wo.weight": "self_attn.o_proj.weight",
+ "attention.wo.bias": "self_attn.o_proj.bias",
+ "attention.q_norm.weight": "self_attn.q_norm.weight",
+ "attention.k_norm.weight": "self_attn.k_norm.weight",
+ "attention_norm.weight": "input_layernorm.weight",
+ "ffn_norm.weight": "post_attention_layernorm.weight",
+ "feed_forward.w1.weight": ("gate_up_proj.weight", 0),
+ "feed_forward.w3.weight": ("gate_up_proj.weight", 1),
+ "feed_forward.w2.weight": "down_proj.weight",
+ }
+ for source_suffix, target in remap.items():
+ if not name.endswith(source_suffix):
+ continue
+ prefix = name[: -len(source_suffix)]
+ if target is None:
+ self._load_fused_qkv(
+ prefix,
+ loaded_weight,
+ params,
+ is_bias=source_suffix.endswith("bias"),
+ )
+ return True
+ if isinstance(target, tuple):
+ target_suffix, shard_id = target
+ else:
+ target_suffix, shard_id = target, None
+ parameter = params[prefix + target_suffix]
+ if shard_id is None:
+ loader = getattr(parameter, "weight_loader", _default_weight_loader)
+ loader(parameter, loaded_weight)
+ else:
+ parameter.weight_loader(parameter, loaded_weight, shard_id)
+ return True
+ return False
+
+ def _load_fused_qkv(
+ self,
+ prefix: str,
+ loaded_weight: Tensor,
+ params: dict[str, nn.Parameter],
+ *,
+ is_bias: bool,
+ ) -> None:
+ target_name = prefix + "self_attn.qkv_proj." + ("bias" if is_bias else "weight")
+ parameter = params[target_name]
+ layer = self.layers[int(prefix.split(".")[1])]
+ q, k, v = loaded_weight.split(
+ [layer.self_attn.q_size, layer.self_attn.kv_size, layer.self_attn.kv_size],
+ dim=0,
+ )
+ for shard_id, weight in (("q", q), ("k", k), ("v", v)):
+ parameter.weight_loader(parameter, weight, shard_id)
+
+
+def _default_weight_loader(parameter: nn.Parameter, loaded_weight: Tensor) -> None:
+ parameter.data.copy_(loaded_weight)
+
+
+EntryClass = Audio8SGLangModel
diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/tokenizer.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/tokenizer.py
new file mode 100644
index 0000000..be786ab
--- /dev/null
+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/tokenizer.py
@@ -0,0 +1,114 @@
+# SPDX-License-Identifier: Apache-2.0
+from __future__ import annotations
+
+import inspect
+import re
+from dataclasses import dataclass
+from typing import Any
+
+import torch
+
+
+@dataclass
+class Reference:
+ text: str
+ vq_codes: torch.Tensor | None = None
+
+
+class Audio8TokenizerAdapter:
+ def __init__(
+ self,
+ tokenizer: Any,
+ *,
+ semantic_begin_id: int,
+ semantic_end_id: int,
+ eos_token_id: int,
+ ) -> None:
+ self._tok = tokenizer
+ self.semantic_begin_id = int(semantic_begin_id)
+ self.semantic_end_id = int(semantic_end_id)
+ self.eos_token_ids = [int(eos_token_id)]
+
+ def _encode(self, text: str) -> torch.Tensor:
+ kwargs: dict[str, Any] = {"add_special_tokens": False}
+ if "allowed_special" in inspect.signature(self._tok.encode).parameters:
+ kwargs["allowed_special"] = "all"
+ return torch.tensor(self._tok.encode(text, **kwargs), dtype=torch.long)
+
+ @staticmethod
+ def _clean(text: str) -> str:
+ return " ".join(str(text).strip().split())
+
+ @classmethod
+ def _reference_text(cls, text: str) -> str:
+ text = cls._clean(text)
+ return text if re.search(r"<\|speaker:\d+\|>", text) else f"<|speaker:0|>{text}"
+
+ def build_prompt(
+ self,
+ text: str,
+ references: list[Reference] | None = None,
+ *,
+ num_codebooks: int = 10,
+ **_: Any,
+ ) -> dict[str, Any]:
+ target = self._clean(text)
+ if not target:
+ raise ValueError("text must not be empty")
+
+ def encode_parts(parts: list[str]) -> torch.Tensor:
+ return torch.cat([self._encode(part) for part in parts])
+
+ if not references:
+ tokens = encode_parts(
+ [
+ "<|im_start|>system\n",
+ "convert the provided text to speech",
+ "<|im_end|>\n",
+ "<|im_start|>user\n",
+ target,
+ "<|im_end|>\n",
+ "<|im_start|>assistant\n<|voice|>",
+ ]
+ )
+ return {
+ "input_ids": tokens,
+ "vq_mask_tokens": torch.zeros(tokens.numel(), dtype=torch.bool),
+ "vq_parts": [],
+ }
+
+ if len(references) != 1:
+ raise ValueError("Audio8 TTS currently supports exactly one reference")
+ reference = references[0]
+ if not reference.text:
+ raise ValueError("reference text is required")
+ if reference.vq_codes is None:
+ raise ValueError("reference VQ codes are required")
+ codes = reference.vq_codes.to(dtype=torch.long, device="cpu")
+ if codes.ndim != 2 or codes.shape[0] != num_codebooks or codes.shape[1] == 0:
+ raise ValueError(
+ f"reference codes must have shape [{num_codebooks}, T>0], got {tuple(codes.shape)}"
+ )
+
+ prefix = encode_parts(
+ [
+ "<|im_start|>system\n",
+ "convert the provided text to speech reference to the following:\n\nText:\n",
+ self._reference_text(reference.text),
+ "\n\nSpeech:\n",
+ ]
+ )
+ semantic = codes[0] + self.semantic_begin_id
+ suffix = encode_parts(
+ [
+ "<|im_end|>\n",
+ "<|im_start|>user\n",
+ target,
+ "<|im_end|>\n",
+ "<|im_start|>assistant\n<|voice|>",
+ ]
+ )
+ tokens = torch.cat((prefix, semantic, suffix))
+ mask = torch.zeros(tokens.numel(), dtype=torch.bool)
+ mask[prefix.numel() : prefix.numel() + semantic.numel()] = True
+ return {"input_ids": tokens, "vq_mask_tokens": mask, "vq_parts": [codes]}
diff --git a/sglang_omni/configs/audio8_tts_0_6b.yaml b/sglang_omni/configs/audio8_tts_0_6b.yaml
new file mode 100644
index 0000000..6d67b19
--- /dev/null
+++ b/sglang_omni/configs/audio8_tts_0_6b.yaml
@@ -0,0 +1,35 @@
+# SPDX-License-Identifier: Apache-2.0
+config_cls: Audio8TTSPipelineConfig
+name: audio8-tts-0.6b
+model_path: /models/Audio8-TTS-Preview-0.6b
+relay_backend: shm
+entry_stage: preprocessing
+stages:
+ - name: preprocessing
+ num_workers: 1
+ executor:
+ factory: sglang_omni.models.audio8_tts.pipeline.stages.create_preprocessing_executor
+ args:
+ device: cuda:0
+ get_next: sglang_omni.models.audio8_tts.pipeline.next_stage.preprocessing_next
+ relay:
+ device: cpu
+ - name: tts_engine
+ num_workers: 1
+ executor:
+ factory: sglang_omni.models.audio8_tts.pipeline.stages.create_sglang_tts_engine_executor
+ args:
+ device: cuda:0
+ max_new_tokens: 1024
+ get_next: sglang_omni.models.audio8_tts.pipeline.next_stage.tts_engine_next
+ relay:
+ device: cuda
+ - name: vocoder
+ num_workers: 1
+ executor:
+ factory: sglang_omni.models.audio8_tts.pipeline.stages.create_vocoder_executor
+ args:
+ device: cuda:0
+ get_next: sglang_omni.models.audio8_tts.pipeline.next_stage.vocoder_next
+ relay:
+ device: cpu
diff --git a/sglang_omni/scripts/install_adapter.sh b/sglang_omni/scripts/install_adapter.sh
new file mode 100755
index 0000000..2a77503
--- /dev/null
+++ b/sglang_omni/scripts/install_adapter.sh
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: Apache-2.0
+set -euo pipefail
+
+if [[ $# -ne 1 ]]; then
+ echo "Usage: $0 <sglang-omni source root or sglang_omni package directory>" >&2
+ exit 2
+fi
+
+BUNDLE_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+SOURCE_DIR="${BUNDLE_ROOT}/adapter/sglang_omni/models/audio8_tts"
+TARGET="$1"
+
+if [[ -d "${TARGET}/sglang_omni/models" ]]; then
+ PACKAGE_DIR="${TARGET}/sglang_omni"
+elif [[ -d "${TARGET}/models" && -f "${TARGET}/__init__.py" ]]; then
+ PACKAGE_DIR="${TARGET}"
+else
+ echo "Cannot find sglang_omni/models under: ${TARGET}" >&2
+ exit 1
+fi
+
+DEST_DIR="${PACKAGE_DIR}/models/audio8_tts"
+mkdir -p "${DEST_DIR}"
+cp -a "${SOURCE_DIR}/." "${DEST_DIR}/"
+
+echo "Installed Audio8 adapter to: ${DEST_DIR}"
+echo "No SGLang-Omni core files were modified."
diff --git a/sglang_omni/scripts/run_server.sh b/sglang_omni/scripts/run_server.sh
new file mode 100755
index 0000000..66b742b
--- /dev/null
+++ b/sglang_omni/scripts/run_server.sh
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: Apache-2.0
+set -euo pipefail
+
+BUNDLE_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+PYTHON_BIN="${PYTHON_BIN:-python3}"
+MODEL="${MODEL:-/models/Audio8-TTS-Preview-0.6b}"
+CONFIG="${CONFIG:-${BUNDLE_ROOT}/configs/audio8_tts_0_6b.yaml}"
+MODEL_NAME="${MODEL_NAME:-audio8/tts-0.6b}"
+HOST="${HOST:-0.0.0.0}"
+PORT="${PORT:-8010}"
+
+if [[ -n "${SGLANG_OMNI_ROOT:-}" ]]; then
+ if [[ -d "${SGLANG_OMNI_ROOT}/sglang_omni" ]]; then
+ export PYTHONPATH="${SGLANG_OMNI_ROOT}${PYTHONPATH:+:${PYTHONPATH}}"
+ elif [[ -f "${SGLANG_OMNI_ROOT}/__init__.py" ]]; then
+ PACKAGE_PARENT="$(cd "${SGLANG_OMNI_ROOT}/.." && pwd)"
+ export PYTHONPATH="${PACKAGE_PARENT}${PYTHONPATH:+:${PYTHONPATH}}"
+ else
+ echo "Invalid SGLANG_OMNI_ROOT: ${SGLANG_OMNI_ROOT}" >&2
+ exit 1
+ fi
+fi
+
+if [[ -n "${SGLANG_OMNI_SITE_PACKAGES:-}" ]]; then
+ if [[ ! -d "${SGLANG_OMNI_SITE_PACKAGES}" ]]; then
+ echo "Invalid SGLANG_OMNI_SITE_PACKAGES: ${SGLANG_OMNI_SITE_PACKAGES}" >&2
+ exit 1
+ fi
+ export PYTHONPATH="${PYTHONPATH:-}${PYTHONPATH:+:}${SGLANG_OMNI_SITE_PACKAGES}"
+fi
+
+export FLASHINFER_WORKSPACE_BASE="${FLASHINFER_WORKSPACE_BASE:-/tmp/audio8-flashinfer}"
+
+exec "${PYTHON_BIN}" -m sglang_omni.cli.cli serve \
+ --model-path "${MODEL}" \
+ --config "${CONFIG}" \
+ --model-name "${MODEL_NAME}" \
+ --host "${HOST}" \
+ --port "${PORT}"
diff --git a/sglang_omni/scripts/smoke_test.sh b/sglang_omni/scripts/smoke_test.sh
new file mode 100755
index 0000000..ed8c2b6
--- /dev/null
+++ b/sglang_omni/scripts/smoke_test.sh
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: Apache-2.0
+set -euo pipefail
+
+BASE_URL="${BASE_URL:-http://127.0.0.1:8010}"
+OUTPUT="${OUTPUT:-/tmp/audio8_sglang_omni_smoke.wav}"
+
+curl -sS \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "model": "audio8/tts-0.6b",
+ "input": "你好,这是 Audio8 TTS 的 SGLang Omni 推理测试。",
+ "response_format": "wav",
+ "max_new_tokens": 128,
+ "temperature": 0.8,
+ "top_p": 0.95,
+ "top_k": 50
+ }' \
+ "${BASE_URL}/v1/audio/speech" \
+ -o "${OUTPUT}"
+
+magic="$(od -An -c -N4 "${OUTPUT}" | tr -d ' ')"
+if [[ "${magic}" != "RIFF" ]]; then
+ echo "Unexpected response header: ${magic}" >&2
+ exit 1
+fi
+wc -c "${OUTPUT}"
diff --git a/sglang_omni/scripts/test_fast_kv_cache.py b/sglang_omni/scripts/test_fast_kv_cache.py
new file mode 100755
index 0000000..91ba259
--- /dev/null
+++ b/sglang_omni/scripts/test_fast_kv_cache.py
@@ -0,0 +1,99 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: Apache-2.0
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from types import SimpleNamespace
+
+import torch
+
+from sglang_omni.models.audio8_tts.sglang_model import FastDecoderLayer, _rope
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--model-path", type=Path, required=True)
+ parser.add_argument("--batch-size", type=int, default=2)
+ parser.add_argument("--seed", type=int, default=1234)
+ args = parser.parse_args()
+
+ config = SimpleNamespace(
+ **json.loads((args.model_path / "config.json").read_text(encoding="utf-8"))
+ )
+ if not torch.cuda.is_available():
+ raise SystemExit("CUDA is required for the Fast AR KV-cache test")
+
+ torch.manual_seed(args.seed)
+ torch.cuda.manual_seed_all(args.seed)
+ device = torch.device("cuda")
+ layers = torch.nn.ModuleList(
+ [FastDecoderLayer(config) for _ in range(config.n_fast_layer)]
+ ).to(device=device, dtype=torch.bfloat16).eval()
+ for layer in layers:
+ layer.attention.setup_audio8_cache(
+ args.batch_size,
+ config.num_codebooks + 1,
+ device=device,
+ dtype=torch.bfloat16,
+ )
+
+ inputs = torch.randn(
+ args.batch_size,
+ config.num_codebooks,
+ config.fast_dim,
+ device=device,
+ dtype=torch.bfloat16,
+ )
+ rope = _rope(
+ config.num_codebooks,
+ config.fast_head_dim,
+ config.rope_base,
+ device,
+ )
+
+ with torch.inference_mode():
+ full = inputs
+ for layer in layers:
+ full = layer(full, rope)
+
+ for layer in layers:
+ layer.attention.clear_audio8_cache()
+ cached_parts = []
+ for position in range(config.num_codebooks):
+ hidden = inputs[:, position : position + 1]
+ cache_positions = torch.full(
+ (args.batch_size,),
+ position,
+ device=device,
+ dtype=torch.int32,
+ )
+ for layer in layers:
+ hidden = layer.forward_audio8_cached(
+ hidden,
+ rope[position : position + 1],
+ cache_positions,
+ )
+ cached_parts.append(hidden)
+ cached = torch.cat(cached_parts, dim=1)
+
+ difference = (full.float() - cached.float()).abs()
+ max_abs = float(difference.max().item())
+ mean_abs = float(difference.mean().item())
+ cosine = float(
+ torch.nn.functional.cosine_similarity(
+ full.float().flatten(),
+ cached.float().flatten(),
+ dim=0,
+ ).item()
+ )
+ print(f"max_abs={max_abs:.6f}")
+ print(f"mean_abs={mean_abs:.6f}")
+ print(f"cosine={cosine:.8f}")
+ if cosine < 0.999:
+ raise SystemExit("Fast KV-cache output diverged from full causal forward")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/sglang_omni/scripts/verify_install.py b/sglang_omni/scripts/verify_install.py
new file mode 100755
index 0000000..9be74d0
--- /dev/null
+++ b/sglang_omni/scripts/verify_install.py
@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: Apache-2.0
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--model-path", type=Path)
+ args = parser.parse_args()
+
+ from sglang_omni.models.audio8_tts.config import Audio8TTSPipelineConfig
+ from sglang_omni.models.audio8_tts.factory import make_server_args
+ from sglang_omni.models.audio8_tts.sglang_model import (
+ Audio8SGLangModel,
+ FastAttention,
+ )
+
+ assert Audio8TTSPipelineConfig.architecture == "ArkttsModel"
+ assert Audio8SGLangModel.__name__ == "Audio8SGLangModel"
+ assert callable(FastAttention.forward_audio8_cached)
+ assert callable(make_server_args)
+
+ if args.model_path is not None:
+ required = {
+ "config.json",
+ "model.safetensors",
+ "codec.pth",
+ "tokenizer.json",
+ "tokenizer_config.json",
+ "modeling_arktts_codec.py",
+ }
+ missing = sorted(name for name in required if not (args.model_path / name).is_file())
+ if missing:
+ raise SystemExit(f"Model directory is missing: {missing}")
+ config = json.loads((args.model_path / "config.json").read_text())
+ if config.get("architectures") != ["ArkttsModel"]:
+ raise SystemExit("config.json architectures must contain ArkttsModel")
+
+ print("Audio8 SGLang-Omni adapter import check passed")
+
+
+if __name__ == "__main__":
+ main()