SOTA-Class TTS at Compact Scale
Python
Add SGLang Omni serving support
linyubupa2b2946e7f046
diff --git a/README.md b/README.mdindex dc91ee7..20f5b19 100644--- a/README.md+++ b/README.md@@ -122,6 +122,127 @@ The batch command writes `manifest.jsonl` and `failures.jsonl`. Existing WAVfiles are skipped unless `--overwrite` is passed. See`python audio8_tts_infer.py --help` for sampling and code-saving options.+## SGLang Omni Serving++The adapter in [`sglang_omni/`](sglang_omni/) provides an OpenAI-compatible+service with SGLang paged attention, dynamic batching, a fixed KV cache for the+fast codebook decoder, reference-audio encoding, and waveform decoding. It is+installed as an independent `audio8_tts` model plugin and does not overwrite+SGLang Omni core files.++### Compatibility++The adapter uses internal SGLang Omni interfaces, so deploy it with the tested+revision instead of the latest `main` branch.++| Dependency | Tested version |+|---|---|+| SGLang Omni | `68a572348837f7b004857b4b07993c20ade4c017` (`0.1.0`) |+| SGLang | `0.5.8` |+| PyTorch | `2.9.1+cu128` |+| Transformers | `4.57.1` |+| Precision | BF16 |++### Install++Run these commands from the Audio8 TTS repository root. The example uses+Python 3.12 and [`uv`](https://docs.astral.sh/uv/).++```bash+export SGLANG_OMNI_ROOT=/opt/sglang-omni+export MODEL=/models/Audio8-TTS-Preview-0.6b++git clone https://github.com/sgl-project/sglang-omni.git "${SGLANG_OMNI_ROOT}"+git -C "${SGLANG_OMNI_ROOT}" checkout 68a572348837f7b004857b4b07993c20ade4c017++uv venv .venv-sglang --python 3.12+source .venv-sglang/bin/activate+uv pip install -v -e "${SGLANG_OMNI_ROOT}"++hf download AutoArk-AI/Audio8-TTS-Preview-0.6b --local-dir "${MODEL}"+./sglang_omni/scripts/install_adapter.sh "${SGLANG_OMNI_ROOT}"+python3 ./sglang_omni/scripts/verify_install.py --model-path "${MODEL}"+```++For an existing wheel or site-packages installation, resolve the package+directory and install the adapter there:++```bash+SGLANG_OMNI_PACKAGE="$(python3 -c 'import importlib.util, pathlib; s=importlib.util.find_spec("sglang_omni"); assert s and s.origin; print(pathlib.Path(s.origin).parent)')"+./sglang_omni/scripts/install_adapter.sh "${SGLANG_OMNI_PACKAGE}"+```++### Start the service++```bash+CUDA_VISIBLE_DEVICES=0 \+SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \+MODEL="${MODEL}" \+HOST=0.0.0.0 \+PORT=8010 \+./sglang_omni/scripts/run_server.sh+```++The defaults use model name `audio8/tts-0.6b`, BF16, one GPU, a `0.2` static+memory fraction, and up to 32 running requests. The main runtime controls are+`MODEL_NAME`, `AUDIO8_TTS_MEM_FRACTION_STATIC`,+`AUDIO8_TTS_MAX_RUNNING_REQUESTS`, `AUDIO8_TTS_CHUNKED_PREFILL_SIZE`, and+`AUDIO8_TTS_DISABLE_CUDA_GRAPH`. Set `SGLANG_OMNI_SITE_PACKAGES` when the+runtime dependencies are installed in a separate site-packages directory.++### Call the API++Generate speech without a reference:++```bash+curl -sS --fail-with-body \+ -H 'Content-Type: application/json' \+ -d '{+ "model": "audio8/tts-0.6b",+ "input": "Hello from Audio8 TTS.",+ "response_format": "wav",+ "max_new_tokens": 256,+ "temperature": 0.8,+ "top_p": 0.95,+ "top_k": 50+ }' \+ http://127.0.0.1:8010/v1/audio/speech \+ -o audio8.wav+```++Generate speech with one reference voice:++```bash+curl -sS --fail-with-body \+ -H 'Content-Type: application/json' \+ -d '{+ "model": "audio8/tts-0.6b",+ "input": "This sentence uses the reference voice.",+ "response_format": "wav",+ "temperature": 0.8,+ "top_p": 0.95,+ "top_k": 50,+ "references": [{+ "audio_path": "/data/reference.wav",+ "text": "The exact transcript of the reference recording."+ }]+ }' \+ http://127.0.0.1:8010/v1/audio/speech \+ -o audio8_clone.wav+```++The reference path must be visible inside the service environment. The current+adapter supports TP=1, one reference per request, and non-streaming WAV output.+Run the smoke test to verify a deployment:++```bash+BASE_URL=http://127.0.0.1:8010 ./sglang_omni/scripts/smoke_test.sh+```++To build the adapter into an existing image, append+[`sglang_omni/Dockerfile.snippet`](sglang_omni/Dockerfile.snippet) after the+SGLang Omni package and its Python dependencies are installed.+## Supervised Fine-tuningInstall the training dependencies first:diff --git a/README_zh.md b/README_zh.mdindex 2b1cf6d..4847dc8 100644--- a/README_zh.md+++ b/README_zh.md@@ -112,6 +112,121 @@ python audio8_tts_infer.py \批量命令会生成 `manifest.jsonl` 和 `failures.jsonl`。除非传入 `--overwrite`,否则已有 WAV 会被跳过。采样和 codec codes 保存参数可通过 `python audio8_tts_infer.py --help` 查看。+## SGLang Omni 服务部署++[`sglang_omni/`](sglang_omni/) 中的适配器提供兼容 OpenAI API 的推理服务,包含 SGLang+paged attention、动态 batching、Fast AR 固定 KV cache、参考音频编码和 waveform 解码。+它以独立的 `audio8_tts` 模型插件安装,不会覆盖 SGLang Omni 核心文件。++### 兼容版本++适配器使用了 SGLang Omni 的内部接口,因此部署时应固定到以下已验证版本,不要直接使用最新+`main` 分支。++| 依赖 | 已验证版本 |+|---|---|+| SGLang Omni | `68a572348837f7b004857b4b07993c20ade4c017`(`0.1.0`) |+| SGLang | `0.5.8` |+| PyTorch | `2.9.1+cu128` |+| Transformers | `4.57.1` |+| 精度 | BF16 |++### 安装++在 Audio8 TTS 仓库根目录执行以下命令。示例使用 Python 3.12 和+[`uv`](https://docs.astral.sh/uv/)。++```bash+export SGLANG_OMNI_ROOT=/opt/sglang-omni+export MODEL=/models/Audio8-TTS-Preview-0.6b++git clone https://github.com/sgl-project/sglang-omni.git "${SGLANG_OMNI_ROOT}"+git -C "${SGLANG_OMNI_ROOT}" checkout 68a572348837f7b004857b4b07993c20ade4c017++uv venv .venv-sglang --python 3.12+source .venv-sglang/bin/activate+uv pip install -v -e "${SGLANG_OMNI_ROOT}"++hf download AutoArk-AI/Audio8-TTS-Preview-0.6b --local-dir "${MODEL}"+./sglang_omni/scripts/install_adapter.sh "${SGLANG_OMNI_ROOT}"+python3 ./sglang_omni/scripts/verify_install.py --model-path "${MODEL}"+```++如果 SGLang Omni 已通过 wheel 或 site-packages 安装,可以定位其包目录后安装适配器:++```bash+SGLANG_OMNI_PACKAGE="$(python3 -c 'import importlib.util, pathlib; s=importlib.util.find_spec("sglang_omni"); assert s and s.origin; print(pathlib.Path(s.origin).parent)')"+./sglang_omni/scripts/install_adapter.sh "${SGLANG_OMNI_PACKAGE}"+```++### 启动服务++```bash+CUDA_VISIBLE_DEVICES=0 \+SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \+MODEL="${MODEL}" \+HOST=0.0.0.0 \+PORT=8010 \+./sglang_omni/scripts/run_server.sh+```++默认配置使用模型名 `audio8/tts-0.6b`、BF16、单卡、`0.2` 静态显存比例和最多 32 个并发+请求。主要运行参数包括 `MODEL_NAME`、`AUDIO8_TTS_MEM_FRACTION_STATIC`、+`AUDIO8_TTS_MAX_RUNNING_REQUESTS`、`AUDIO8_TTS_CHUNKED_PREFILL_SIZE` 和+`AUDIO8_TTS_DISABLE_CUDA_GRAPH`。如果运行依赖安装在单独的 site-packages 目录中,请设置+`SGLANG_OMNI_SITE_PACKAGES`。++### 调用 API++无参考音频生成:++```bash+curl -sS --fail-with-body \+ -H 'Content-Type: application/json' \+ -d '{+ "model": "audio8/tts-0.6b",+ "input": "你好,这是 Audio8 TTS 服务测试。",+ "response_format": "wav",+ "max_new_tokens": 256,+ "temperature": 0.8,+ "top_p": 0.95,+ "top_k": 50+ }' \+ http://127.0.0.1:8010/v1/audio/speech \+ -o audio8.wav+```++使用一个参考音色生成:++```bash+curl -sS --fail-with-body \+ -H 'Content-Type: application/json' \+ -d '{+ "model": "audio8/tts-0.6b",+ "input": "这句话使用参考音色生成。",+ "response_format": "wav",+ "temperature": 0.8,+ "top_p": 0.95,+ "top_k": 50,+ "references": [{+ "audio_path": "/data/reference.wav",+ "text": "参考录音对应的准确文本。"+ }]+ }' \+ http://127.0.0.1:8010/v1/audio/speech \+ -o audio8_clone.wav+```++参考音频路径必须在服务运行环境中可见。当前适配器支持 TP=1、每个请求一个 reference 和+非流式 WAV 输出。运行以下 smoke test 可以验证部署:++```bash+BASE_URL=http://127.0.0.1:8010 ./sglang_omni/scripts/smoke_test.sh+```++构建镜像时,请在安装 SGLang Omni 包及其 Python 依赖后追加+[`sglang_omni/Dockerfile.snippet`](sglang_omni/Dockerfile.snippet)。+## SFT 训练先安装训练依赖:diff --git a/sglang_omni/Dockerfile.snippet b/sglang_omni/Dockerfile.snippetnew file mode 100644index 0000000..5ffd879--- /dev/null+++ b/sglang_omni/Dockerfile.snippet@@ -0,0 +1,22 @@+# SPDX-License-Identifier: Apache-2.0+# Add these instructions to the SGLang-Omni image after its Python environment+# and sglang_omni package have been installed.++COPY sglang_omni /opt/audio8-adapter++# This works for both wheel and editable installs. It resolves the package+# without importing the full SGLang runtime during docker build.+RUN SGLANG_OMNI_PACKAGE="$(python3 -c 'import importlib.util, pathlib; s=importlib.util.find_spec("sglang_omni"); assert s and s.origin; print(pathlib.Path(s.origin).parent)')" && \+ /opt/audio8-adapter/scripts/install_adapter.sh "${SGLANG_OMNI_PACKAGE}" && \+ python3 /opt/audio8-adapter/scripts/verify_install.py++ENV AUDIO8_TTS_MEM_FRACTION_STATIC=0.2 \+ AUDIO8_TTS_MAX_RUNNING_REQUESTS=32 \+ AUDIO8_TTS_DISABLE_CUDA_GRAPH=0 \+ FLASHINFER_WORKSPACE_BASE=/tmp/audio8-flashinfer++# Mount or COPY the checkpoint to this path at runtime:+# /models/Audio8-TTS-Preview-0.6b+#+# Example container command:+# /opt/audio8-adapter/scripts/run_server.shdiff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/__init__.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/__init__.pynew file mode 100644index 0000000..56bc896--- /dev/null+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/__init__.py@@ -0,0 +1,6 @@+# SPDX-License-Identifier: Apache-2.0+"""Audio8 TTS support for SGLang-Omni."""++from . import config++__all__ = ["config"]diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/config.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/config.pynew file mode 100644index 0000000..5078e63--- /dev/null+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/config.py@@ -0,0 +1,47 @@+# SPDX-License-Identifier: Apache-2.0+from __future__ import annotations++from typing import ClassVar++from sglang_omni.config import ExecutorConfig, PipelineConfig, RelayConfig, StageConfig++_PKG = "sglang_omni.models.audio8_tts.pipeline"+++class Audio8TTSPipelineConfig(PipelineConfig):+ architecture: ClassVar[str] = "ArkttsModel"++ model_path: str+ entry_stage: str = "preprocessing"+ stages: list[StageConfig] = [+ StageConfig(+ name="preprocessing",+ executor=ExecutorConfig(+ factory=f"{_PKG}.stages.create_preprocessing_executor",+ args={"device": "cuda:0"},+ ),+ get_next=f"{_PKG}.next_stage.preprocessing_next",+ relay=RelayConfig(device="cpu"),+ ),+ StageConfig(+ name="tts_engine",+ executor=ExecutorConfig(+ factory=f"{_PKG}.stages.create_sglang_tts_engine_executor",+ args={"device": "cuda:0", "max_new_tokens": 1024},+ ),+ get_next=f"{_PKG}.next_stage.tts_engine_next",+ relay=RelayConfig(device="cuda"),+ ),+ StageConfig(+ name="vocoder",+ executor=ExecutorConfig(+ factory=f"{_PKG}.stages.create_vocoder_executor",+ args={"device": "cuda:0"},+ ),+ get_next=f"{_PKG}.next_stage.vocoder_next",+ relay=RelayConfig(device="cpu"),+ ),+ ]+++EntryClass = Audio8TTSPipelineConfigdiff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.pynew file mode 100644index 0000000..ef04805--- /dev/null+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/factory.py@@ -0,0 +1,120 @@+# SPDX-License-Identifier: Apache-2.0+from __future__ import annotations++import os+from typing import Any++from sglang_omni.engines.omni.engine import OmniEngine+from sglang_omni.engines.omni.scheduler import Scheduler+from sglang_omni.models.audio8_tts.runtime.audio8_sglang_ar import (+ Audio8IterationController,+ Audio8ModelRunner,+ Audio8ResourceManager,+)+++def create_audio8_engine(+ server_args: Any,+ *,+ gpu_id: int,+ eos_token_id: int,+ max_new_tokens: int,+) -> OmniEngine:+ from sglang.srt.models.registry import ModelRegistry++ from sglang_omni.engines.ar.sglang_backend.model_worker import (+ ModelWorker,+ ModelWorkerConfig,+ )+ from sglang_omni.engines.ar.sglang_backend.scheduler.cache import create_tree_cache+ from sglang_omni.engines.ar.sglang_backend.scheduler.decode import DecodeManager+ from sglang_omni.engines.ar.sglang_backend.scheduler.prefill import PrefillManager+ from sglang_omni.engines.omni.runtime.sglang_ar import SGLangBatchPlanner+ from sglang_omni.models.audio8_tts.sglang_model import Audio8SGLangModel++ # Register lazily in the worker process so the adapter does not require a+ # patch to SGLang-Omni's hard-coded model registry bootstrap.+ ModelRegistry.models["ArkttsModel"] = Audio8SGLangModel++ if server_args.attention_backend is None:+ server_args.attention_backend = "fa3"+ disable_attr = "disable_" + "cuda_graph"+ want_cuda_graph = not bool(getattr(server_args, disable_attr, False))+ setattr(server_args, disable_attr, True)+ model_worker = ModelWorker(+ config=ModelWorkerConfig(),+ server_args=server_args,+ gpu_id=gpu_id,+ )+ model_worker.model_runner.model.setup_audio8_decode(+ server_args.max_running_requests+ )+ setattr(server_args, disable_attr, not want_cuda_graph)+ if want_cuda_graph:+ model_worker.model_runner.init_device_graphs()++ request_pool, kv_allocator = model_worker.get_memory_pool()+ tree_cache = create_tree_cache(+ server_args,+ request_pool,+ kv_allocator,+ server_args.page_size,+ )+ prefill_manager = PrefillManager(+ page_size=server_args.page_size,+ chunked_prefill_size=server_args.chunked_prefill_size,+ max_prefill_tokens=server_args.max_prefill_tokens,+ req_to_token_pool=request_pool,+ token_to_kv_pool_allocator=kv_allocator,+ tree_cache=tree_cache,+ model_config=model_worker.model_config,+ enable_overlap=False,+ )+ decode_manager = DecodeManager(+ server_args=server_args,+ token_to_kv_pool_allocator=kv_allocator,+ on_retract=lambda request: prefill_manager.add_one_request(request),+ )+ planner = SGLangBatchPlanner(prefill_manager, decode_manager, server_args)+ scheduler = Scheduler(+ batch_planner=planner,+ resource_manager=Audio8ResourceManager(+ kv_allocator,+ request_pool,+ tree_cache,+ ),+ iteration_controller=Audio8IterationController(+ tree_cache,+ eos_token_id=eos_token_id,+ max_new_tokens=max_new_tokens,+ ),+ stream_adapter=lambda request, output: (+ None if output.data is None else output.data.codes+ ),+ )+ return OmniEngine(+ scheduler=scheduler,+ model_runner=Audio8ModelRunner(model_worker, planner),+ enable_overlap=False,+ )+++def make_server_args(model_path: str) -> Any:+ from sglang.srt.server_args import ServerArgs++ server_args = ServerArgs(+ model_path=model_path,+ tp_size=1,+ dtype="bfloat16",+ trust_remote_code=True,+ mem_fraction_static=float(os.getenv("AUDIO8_TTS_MEM_FRACTION_STATIC", "0.2")),+ chunked_prefill_size=int(os.getenv("AUDIO8_TTS_CHUNKED_PREFILL_SIZE", "2048")),+ max_running_requests=int(os.getenv("AUDIO8_TTS_MAX_RUNNING_REQUESTS", "32")),+ disable_radix_cache=os.getenv("AUDIO8_TTS_DISABLE_RADIX_CACHE", "1") == "1",+ )+ setattr(+ server_args,+ "disable_" + "cuda_graph",+ os.getenv("AUDIO8_TTS_DISABLE_CUDA_GRAPH", "0") == "1",+ )+ return server_argsdiff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/io.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/io.pynew file mode 100644index 0000000..0b3a5e6--- /dev/null+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/io.py@@ -0,0 +1,82 @@+# SPDX-License-Identifier: Apache-2.0+from __future__ import annotations++from dataclasses import dataclass+from typing import Any++import torch+++@dataclass+class Audio8TTSState:+ input_ids: Any = None+ vq_mask_tokens: Any | None = None+ vq_parts: Any | None = None+ num_codebooks: int = 10+ codebook_size: int = 4096+ max_new_tokens: int = 1024+ temperature: float = 0.8+ top_p: float = 0.95+ top_k: int = 50+ do_sample: bool = True+ output_codes: Any | None = None+ prompt_tokens: int = 0+ completion_tokens: int = 0+ audio_samples: Any | None = None+ sample_rate: int = 44100++ @staticmethod+ def _to_list(value: Any) -> Any:+ return value.tolist() if isinstance(value, torch.Tensor) else value++ def to_dict(self) -> dict[str, Any]:+ data = {+ "input_ids": self._to_list(self.input_ids),+ "vq_mask_tokens": self._to_list(self.vq_mask_tokens),+ "vq_parts": (+ [self._to_list(part) for part in self.vq_parts]+ if self.vq_parts is not None+ else None+ ),+ "num_codebooks": self.num_codebooks,+ "codebook_size": self.codebook_size,+ "max_new_tokens": self.max_new_tokens,+ "temperature": self.temperature,+ "top_p": self.top_p,+ "top_k": self.top_k,+ "do_sample": self.do_sample,+ "output_codes": self._to_list(self.output_codes),+ "prompt_tokens": self.prompt_tokens,+ "completion_tokens": self.completion_tokens,+ "audio_samples": self._to_list(self.audio_samples),+ "sample_rate": self.sample_rate,+ }+ return {key: value for key, value in data.items() if value is not None}++ @classmethod+ def from_dict(cls, data: dict[str, Any]) -> "Audio8TTSState":+ vq_parts = data.get("vq_parts")+ if vq_parts is not None:+ vq_parts = [torch.as_tensor(part, dtype=torch.long) for part in vq_parts]+ output_codes = data.get("output_codes")+ return cls(+ input_ids=data.get("input_ids"),+ vq_mask_tokens=data.get("vq_mask_tokens"),+ vq_parts=vq_parts,+ num_codebooks=int(data.get("num_codebooks", 10)),+ codebook_size=int(data.get("codebook_size", 4096)),+ max_new_tokens=int(data.get("max_new_tokens", 1024)),+ temperature=float(data.get("temperature", 0.8)),+ top_p=float(data.get("top_p", 0.95)),+ top_k=int(data.get("top_k", 50)),+ do_sample=bool(data.get("do_sample", True)),+ output_codes=(+ torch.as_tensor(output_codes, dtype=torch.long)+ if output_codes is not None+ else None+ ),+ prompt_tokens=int(data.get("prompt_tokens", 0)),+ completion_tokens=int(data.get("completion_tokens", 0)),+ audio_samples=data.get("audio_samples"),+ sample_rate=int(data.get("sample_rate", 44100)),+ )diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/__init__.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/__init__.pynew file mode 100644index 0000000..501ad20--- /dev/null+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/__init__.py@@ -0,0 +1,2 @@+# SPDX-License-Identifier: Apache-2.0+"""Audio8 TTS pipeline stages."""diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/engine_io.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/engine_io.pynew file mode 100644index 0000000..704c780--- /dev/null+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/engine_io.py@@ -0,0 +1,64 @@+# SPDX-License-Identifier: Apache-2.0+from __future__ import annotations++from typing import Any++import torch++from sglang_omni.models.audio8_tts.io import Audio8TTSState+from sglang_omni.models.audio8_tts.runtime.audio8_sglang_ar import (+ Audio8SGLangRequestData,+)+++def build_tts_request(+ state: Audio8TTSState,+ tokenizer: Any,+ request_id: str,+) -> Audio8SGLangRequestData:+ from sglang.srt.managers.schedule_batch import Req+ from sglang.srt.sampling.sampling_params import SamplingParams++ input_ids = list(state.input_ids)+ sampling_params = SamplingParams(+ max_new_tokens=state.max_new_tokens,+ temperature=state.temperature,+ )+ req = Req(+ rid=request_id,+ origin_input_text="",+ origin_input_ids=input_ids,+ sampling_params=sampling_params,+ vocab_size=tokenizer.vocab_size,+ )+ return Audio8SGLangRequestData(+ input_ids=torch.tensor(input_ids, dtype=torch.long),+ req=req,+ vq_mask_tokens=(+ torch.as_tensor(state.vq_mask_tokens, dtype=torch.bool)+ if state.vq_mask_tokens is not None+ else None+ ),+ vq_parts=(+ [torch.as_tensor(part, dtype=torch.long) for part in state.vq_parts]+ if state.vq_parts is not None+ else None+ ),+ num_codebooks=state.num_codebooks,+ codebook_size=state.codebook_size,+ max_new_tokens=state.max_new_tokens,+ temperature=state.temperature,+ top_p=state.top_p,+ top_k=state.top_k,+ do_sample=state.do_sample,+ )+++def apply_tts_result(state: Audio8TTSState, result: Audio8SGLangRequestData) -> None:+ if result.output_codes:+ all_codes = torch.cat(result.output_codes, dim=1)+ state.output_codes = all_codes[1:]+ state.completion_tokens = int(all_codes.shape[1])+ else:+ state.output_codes = None+ state.prompt_tokens = len(result.input_ids) if result.input_ids is not None else 0diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/next_stage.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/next_stage.pynew file mode 100644index 0000000..3ce60c0--- /dev/null+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/next_stage.py@@ -0,0 +1,17 @@+# SPDX-License-Identifier: Apache-2.0+from typing import Any+++def preprocessing_next(request_id: str, output: Any) -> str:+ del request_id, output+ return "tts_engine"+++def tts_engine_next(request_id: str, output: Any) -> str:+ del request_id, output+ return "vocoder"+++def vocoder_next(request_id: str, output: Any) -> None:+ del request_id, output+ return Nonediff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/stages.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/stages.pynew file mode 100644index 0000000..c69c496--- /dev/null+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/stages.py@@ -0,0 +1,261 @@+# SPDX-License-Identifier: Apache-2.0+from __future__ import annotations++import base64+import importlib.util+import json+import logging+import sys+from pathlib import Path+from types import SimpleNamespace+from typing import Any+from urllib.parse import unquote_to_bytes++import torch++from sglang_omni.executors import EngineExecutor, PreprocessingExecutor+from sglang_omni.models.audio8_tts.factory import create_audio8_engine, make_server_args+from sglang_omni.models.audio8_tts.io import Audio8TTSState+from sglang_omni.models.audio8_tts.pipeline.engine_io import (+ apply_tts_result,+ build_tts_request,+)+from sglang_omni.models.audio8_tts.pipeline.state_io import load_state, store_state+from sglang_omni.models.audio8_tts.tokenizer import Audio8TokenizerAdapter, Reference+from sglang_omni.proto import StagePayload++logger = logging.getLogger(__name__)+++def _load_config(model_path: str) -> SimpleNamespace:+ with open(Path(model_path) / "config.json", encoding="utf-8") as handle:+ return SimpleNamespace(**json.load(handle))+++def _load_codec(model_path: str, device: str) -> Any:+ model_dir = Path(model_path)+ module_path = model_dir / "modeling_arktts_codec.py"+ spec = importlib.util.spec_from_file_location("_audio8_arktts_codec", module_path)+ if spec is None or spec.loader is None:+ raise ImportError(f"Cannot import Audio8 codec from {module_path}")+ module = importlib.util.module_from_spec(spec)+ sys.modules[spec.name] = module+ spec.loader.exec_module(module)+ codec = module.ArkttsCodec(_load_config(model_path))+ state = torch.load(+ model_dir / "codec.pth",+ map_location="cpu",+ weights_only=True,+ mmap=True,+ )+ if "state_dict" in state:+ state = state["state_dict"]+ if any("generator." in key for key in state):+ state = {+ key.replace("generator.", ""): value+ for key, value in state.items()+ if "generator." in key+ }+ state = {+ key: value+ for key, value in state.items()+ if not key.endswith(("freqs_cis", "causal_mask"))+ }+ codec.load_state_dict(state, strict=True, assign=True)+ dtype = torch.float32 if device == "cpu" else torch.bfloat16+ return codec.eval().to(device=device, dtype=dtype)+++def _load_audio(audio_path: str, sample_rate: int) -> torch.Tensor:+ import httpx++ from sglang_omni.preprocessing.audio import AudioMediaIO++ media_io = AudioMediaIO(target_sr=sample_rate)+ if audio_path.startswith("data:"):+ header, payload = audio_path.split(",", 1)+ raw = base64.b64decode(payload) if ";base64" in header else unquote_to_bytes(payload)+ audio, _ = media_io.load_bytes(raw)+ elif audio_path.startswith(("http://", "https://")):+ response = httpx.get(audio_path, follow_redirects=True, timeout=30)+ response.raise_for_status()+ audio, _ = media_io.load_bytes(response.content)+ else:+ audio, _ = media_io.load_file(Path(audio_path))+ value = torch.from_numpy(audio).float()+ if value.ndim == 1:+ value = value.unsqueeze(0)+ elif value.ndim == 2 and value.shape[0] > 1:+ value = value.mean(dim=0, keepdim=True)+ return value+++def _encode_reference(codec: Any, audio_path: str, device: str) -> torch.Tensor:+ audio = _load_audio(audio_path, codec.sample_rate).to(+ device=device,+ dtype=next(codec.parameters()).dtype,+ )+ lengths = torch.tensor([audio.shape[-1]], device=device, dtype=torch.long)+ with torch.inference_mode():+ codes, code_lengths = codec.encode(audio, lengths)+ return codes[0, :, : int(code_lengths[0].item())].cpu()+++def _validate_codes(value: Any, num_codebooks: int, codebook_size: int) -> torch.Tensor:+ codes = torch.as_tensor(value, dtype=torch.long, device="cpu")+ if codes.ndim != 2 or codes.shape[0] != num_codebooks or codes.shape[1] == 0:+ raise ValueError(+ f"reference codes must have shape [{num_codebooks}, T>0], got {tuple(codes.shape)}"+ )+ if int(codes.min().item()) < 0 or int(codes.max().item()) >= codebook_size:+ raise ValueError(f"reference codes must be in [0, {codebook_size - 1}]")+ return codes+++def create_preprocessing_executor(+ model_path: str,+ *,+ device: str = "cuda:0",+) -> PreprocessingExecutor:+ from transformers import PreTrainedTokenizerFast++ config = _load_config(model_path)+ tokenizer = PreTrainedTokenizerFast.from_pretrained(+ model_path,+ fix_mistral_regex=False,+ )+ adapter = Audio8TokenizerAdapter(+ tokenizer,+ semantic_begin_id=config.semantic_begin_id,+ semantic_end_id=config.semantic_end_id,+ eos_token_id=config.eos_token_id,+ )+ codec: Any | None = None++ def get_codec() -> Any:+ nonlocal codec+ if codec is None:+ logger.info("Loading Audio8 reference codec on %s", device)+ codec = _load_codec(model_path, device)+ return codec++ def preprocess(payload: StagePayload) -> StagePayload:+ inputs = payload.request.inputs or {}+ params = payload.request.params or {}+ if isinstance(inputs, str):+ inputs = {"text": inputs}+ references: list[Reference] | None = None+ raw_references = inputs.get("references")+ if raw_references:+ references = []+ for raw in raw_references:+ codes = raw.get("vq_codes")+ if codes is None:+ audio_path = raw.get("audio_path")+ if not audio_path:+ raise ValueError("reference requires audio_path or vq_codes")+ codes = _encode_reference(get_codec(), audio_path, device)+ codes = _validate_codes(codes, config.num_codebooks, config.codebook_size)+ references.append(Reference(text=raw.get("text", ""), vq_codes=codes))+ prompt = adapter.build_prompt(+ inputs.get("text", ""),+ references,+ num_codebooks=config.num_codebooks,+ )+ state = Audio8TTSState(+ input_ids=prompt["input_ids"],+ vq_mask_tokens=prompt["vq_mask_tokens"],+ vq_parts=prompt["vq_parts"],+ num_codebooks=config.num_codebooks,+ codebook_size=config.codebook_size,+ max_new_tokens=int(params.get("max_new_tokens", 1024)),+ temperature=float(params.get("temperature", 0.8)),+ top_p=float(params.get("top_p", 0.95)),+ top_k=int(params.get("top_k", 50)),+ do_sample=float(params.get("temperature", 0.8)) > 0,+ sample_rate=config.codec_sample_rate,+ )+ return store_state(payload, state)++ return PreprocessingExecutor(preprocess)+++def create_sglang_tts_engine_executor(+ model_path: str,+ *,+ device: str = "cuda:0",+ max_new_tokens: int = 1024,+) -> EngineExecutor:+ from transformers import PreTrainedTokenizerFast++ config = _load_config(model_path)+ tokenizer = PreTrainedTokenizerFast.from_pretrained(+ model_path,+ fix_mistral_regex=False,+ )+ server_args = make_server_args(model_path)+ gpu_id = int(device.split(":")[-1]) if ":" in device else 0+ engine = create_audio8_engine(+ server_args,+ gpu_id=gpu_id,+ eos_token_id=config.eos_token_id,+ max_new_tokens=max_new_tokens,+ )++ def request_builder(payload: StagePayload):+ return build_tts_request(load_state(payload), tokenizer, payload.request_id)++ def result_builder(payload: StagePayload, result: Any) -> StagePayload:+ state = load_state(payload)+ apply_tts_result(state, result)+ return store_state(payload, state)++ return EngineExecutor(+ engine=engine,+ request_builder=request_builder,+ result_builder=result_builder,+ )+++def create_vocoder_executor(+ model_path: str,+ *,+ device: str = "cuda:0",+) -> PreprocessingExecutor:+ codec = _load_codec(model_path, device)+ config = _load_config(model_path)+ warmup_codes = torch.zeros(+ 1,+ config.num_codebooks,+ 4,+ device=device,+ dtype=torch.long,+ )+ with torch.inference_mode():+ codec.decode(warmup_codes)++ def vocode(payload: StagePayload) -> StagePayload:+ state = load_state(payload)+ if state.output_codes is None:+ raise ValueError("Audio8 generation produced no codec frames")+ codes = state.output_codes.to(device=device, dtype=torch.long)+ with torch.inference_mode():+ audio = codec.decode(codes.unsqueeze(0))[0, 0].float().cpu()+ state.audio_samples = audio+ state.sample_rate = codec.sample_rate+ payload = store_state(payload, state)+ payload.data.update(+ {+ "audio_data": audio.tolist(),+ "sample_rate": codec.sample_rate,+ "modality": "audio",+ "usage": {+ "prompt_tokens": state.prompt_tokens,+ "completion_tokens": state.completion_tokens,+ "total_tokens": state.prompt_tokens + state.completion_tokens,+ },+ }+ )+ return payload++ return PreprocessingExecutor(vocode)diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/state_io.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/state_io.pynew file mode 100644index 0000000..3607bbd--- /dev/null+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/pipeline/state_io.py@@ -0,0 +1,12 @@+# SPDX-License-Identifier: Apache-2.0+from sglang_omni.models.audio8_tts.io import Audio8TTSState+from sglang_omni.proto import StagePayload+++def load_state(payload: StagePayload) -> Audio8TTSState:+ return Audio8TTSState.from_dict(payload.data)+++def store_state(payload: StagePayload, state: Audio8TTSState) -> StagePayload:+ payload.data = state.to_dict()+ return payloaddiff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/runtime/__init__.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/runtime/__init__.pynew file mode 100644index 0000000..8b2f235--- /dev/null+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/runtime/__init__.py@@ -0,0 +1,4 @@+# SPDX-License-Identifier: Apache-2.0+from .audio8_sglang_ar import Audio8SGLangRequestData, Audio8StepOutput++__all__ = ["Audio8SGLangRequestData", "Audio8StepOutput"]diff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/runtime/audio8_sglang_ar.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/runtime/audio8_sglang_ar.pynew file mode 100644index 0000000..a6e036d--- /dev/null+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/runtime/audio8_sglang_ar.py@@ -0,0 +1,226 @@+# SPDX-License-Identifier: Apache-2.0+from __future__ import annotations++from dataclasses import dataclass, field+from typing import TYPE_CHECKING, Any++import torch+from sglang.srt.mem_cache.common import release_kv_cache++from sglang_omni.engines.omni.runtime.sglang_ar import (+ SGLangARRequestData,+ SGLangBatchPlanner,+ SGLangResourceManager,+)+from sglang_omni.engines.omni.types import (+ ModelRunnerOutput,+ RequestOutput,+ SchedulerOutput,+ SchedulerRequest,+)++if TYPE_CHECKING:+ from sglang_omni.engines.ar.sglang_backend.model_worker import ModelWorker+++@dataclass+class Audio8StepOutput:+ codes: torch.Tensor+++@dataclass+class Audio8SGLangRequestData(SGLangARRequestData):+ vq_mask_tokens: torch.Tensor | None = None+ vq_parts: list[torch.Tensor] | None = None+ num_codebooks: int = 10+ codebook_size: int = 4096+ output_codes: list[torch.Tensor] = field(default_factory=list)+ max_new_tokens: int | None = None+ temperature: float = 0.8+ top_p: float = 0.95+ top_k: int = 50+ do_sample: bool = True+ _previous_semantic_tokens: list[int] = field(default_factory=list)+ _last_codebook_values: torch.Tensor | None = None+++class Audio8IterationController:+ def __init__(self, tree_cache: Any, eos_token_id: int, max_new_tokens: int) -> None:+ self.tree_cache = tree_cache+ self.eos_token_id = int(eos_token_id)+ self.max_new_tokens = int(max_new_tokens)++ def update_request(self, request: SchedulerRequest, output: RequestOutput) -> None:+ data: Audio8SGLangRequestData = request.data+ req = data.req+ if req.is_chunked > 0:+ output.data = None+ req.is_chunked -= 1+ return+ codes = output.data.codes.clone()+ semantic = int(codes[0, -1].item())+ if semantic != self.eos_token_id:+ # HF generation does not send the EOS step to the waveform codec.+ if data.output_codes:+ data._previous_semantic_tokens.append(semantic)+ data.output_codes.append(codes)+ data._last_codebook_values = codes[1:, 0].clone()+ req.output_ids.append(semantic)+ if not req.finished() and req.decode_batch_idx == 0:+ self.tree_cache.cache_unfinished_req(req)++ def is_finished(self, request: SchedulerRequest, output: RequestOutput) -> bool:+ data: Audio8SGLangRequestData = request.data+ if data.req.is_chunked > 0:+ return False+ semantic = int(output.data.codes[0, -1].item())+ if semantic == self.eos_token_id:+ return True+ return len(data.output_codes) >= (data.max_new_tokens or self.max_new_tokens)+++class Audio8ModelRunner:+ def __init__(self, model_worker: "ModelWorker", batch_planner: SGLangBatchPlanner):+ self.model_worker = model_worker+ self.batch_planner = batch_planner++ def _inject_reference_embeds(+ self,+ model_worker_batch: Any,+ scheduler_output: SchedulerOutput,+ ) -> None:+ device = model_worker_batch.input_ids.device+ model = self.model_worker.model_runner.model+ text_embeds = model.get_embed_tokens()(model_worker_batch.input_ids)+ offset = 0+ for scheduled in scheduler_output.requests:+ data: Audio8SGLangRequestData = scheduled.data+ request_length = int(data.req.extend_input_len)+ prefix_length = len(data.req.prefix_indices)+ if data.vq_mask_tokens is not None and data.vq_parts:+ mask = data.vq_mask_tokens.to(device=device, dtype=torch.bool).flatten()+ expected_length = prefix_length + request_length+ if mask.numel() != expected_length:+ raise ValueError(+ f"Audio8 reference mask length {mask.numel()} != request length {expected_length}"+ )+ mask_slice = mask[prefix_length : prefix_length + request_length]+ parts = [part.to(device=device, dtype=torch.long).T for part in data.vq_parts]+ all_codes = torch.cat(parts, dim=0)+ if all_codes.shape[0] != int(mask.sum().item()):+ raise ValueError("Audio8 reference mask/code length mismatch")+ before = int(mask[:prefix_length].sum().item())+ count = int(mask_slice.sum().item())+ if count:+ codes = all_codes[before : before + count]+ if codes.shape[1] != model.config.num_codebooks:+ raise ValueError("Audio8 reference has the wrong number of codebooks")+ if int(codes.min().item()) < 0 or int(codes.max().item()) >= model.config.codebook_size:+ raise ValueError("Audio8 reference code is outside the embedding range")+ embedded_codes = codes + model._codebook_offsets[None]+ codebook_sum = model.codebook_embeddings(embedded_codes).sum(dim=1)+ indices = mask_slice.nonzero(as_tuple=True)[0] + offset+ text_embeds[indices] += codebook_sum.to(text_embeds.dtype)+ offset += request_length+ model_worker_batch.input_embeds = text_embeds++ def _update_runtime_buffers(+ self,+ model_worker_batch: Any,+ scheduler_output: SchedulerOutput,+ *,+ is_prefill: bool,+ ) -> None:+ model = self.model_worker.model_runner.model+ batch = len(scheduler_output.requests)+ model._previous_valid[:batch].zero_()+ if not is_prefill:+ input_ids = model_worker_batch.input_ids+ semantic_mask = (input_ids >= model.config.semantic_begin_id) & (+ input_ids <= model.config.semantic_end_id+ )+ model._vq_mask[:batch].copy_(semantic_mask)+ else:+ model._vq_mask[:batch].zero_()++ for index, scheduled in enumerate(scheduler_output.requests):+ data: Audio8SGLangRequestData = scheduled.data+ model._temperature[index] = max(float(data.temperature), 1e-5)+ model._top_p[index] = min(max(float(data.top_p), 1e-5), 1.0)+ model._top_k[index] = min(+ max(int(data.top_k), 1),+ int(model.config.codebook_size),+ )+ model._do_sample[index] = bool(data.do_sample)+ history = data._previous_semantic_tokens[-model.config.ras_window_size :]+ if history:+ length = len(history)+ model._previous_semantic[index, -length:] = torch.tensor(+ history,+ device=model._previous_semantic.device,+ dtype=torch.long,+ )+ model._previous_valid[index, -length:] = True+ if (+ not is_prefill+ and data._last_codebook_values is not None+ and bool(model._vq_mask[index].item())+ ):+ model._vq_codes[index].copy_(data._last_codebook_values)++ def _build_outputs(self, scheduler_output: SchedulerOutput) -> dict[str, RequestOutput]:+ model = self.model_worker.model_runner.model+ outputs: dict[str, RequestOutput] = {}+ for index, scheduled in enumerate(scheduler_output.requests):+ data: Audio8SGLangRequestData = scheduled.data+ outputs[scheduled.request_id] = RequestOutput(+ request_id=scheduled.request_id,+ data=(+ None+ if data.req.is_chunked > 0+ else Audio8StepOutput(model._output_codes[index].unsqueeze(-1))+ ),+ finished=False,+ )+ return outputs++ def execute(self, scheduler_output: SchedulerOutput) -> ModelRunnerOutput:+ from sglang.srt.model_executor.forward_batch_info import ForwardBatch++ schedule_batch = scheduler_output.batch_data+ worker_batch = schedule_batch.get_model_worker_batch()+ is_prefill = schedule_batch.forward_mode.is_extend()+ self._update_runtime_buffers(+ worker_batch,+ scheduler_output,+ is_prefill=is_prefill,+ )+ if is_prefill:+ self._inject_reference_embeds(worker_batch, scheduler_output)+ forward_batch = ForwardBatch.init_new(worker_batch, self.model_worker.model_runner)+ batch_result = self.model_worker.forward_batch_generation(forward_batch)+ if schedule_batch.is_prefill_only:+ batch_result.next_token_ids = torch.zeros(+ len(worker_batch.seq_lens),+ dtype=torch.long,+ device=worker_batch.input_ids.device,+ )+ self.batch_planner.record_last_batch(schedule_batch)+ outputs = self._build_outputs(scheduler_output)+ model = self.model_worker.model_runner.model+ batch = len(scheduler_output.requests)+ schedule_batch.output_ids = model._output_semantic_ids[:batch].clone()+ request_ids = [request.request_id for request in scheduler_output.requests]+ return ModelRunnerOutput(+ outputs=outputs,+ req_ids=request_ids,+ req_id_to_index={request_id: index for index, request_id in enumerate(request_ids)},+ )+++class Audio8ResourceManager(SGLangResourceManager):+ def free(self, request: SchedulerRequest) -> None:+ data: Audio8SGLangRequestData = request.data+ release_kv_cache(data.req, self.tree_cache)+ data._previous_semantic_tokens.clear()+ data._last_codebook_values = Nonediff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/sglang_model.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/sglang_model.pynew file mode 100644index 0000000..986432f--- /dev/null+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/sglang_model.py@@ -0,0 +1,771 @@+# SPDX-License-Identifier: Apache-2.0+"""SGLang-native slow AR plus Audio8's fixed-length fast codebook head."""++from __future__ import annotations++import logging+from typing import Any, Iterable, Optional, Tuple++import torch+import torch.nn.functional as F+from sgl_kernel.flash_attn import flash_attn_with_kvcache+from sglang.srt.layers.logits_processor import LogitsProcessorOutput+from torch import Tensor, nn++from sglang_omni.vendor.sglang.core import ForwardBatch+from sglang_omni.vendor.sglang.layers import (+ MergedColumnParallelLinear,+ QKVParallelLinear,+ RadixAttention,+ RMSNorm,+ RowParallelLinear,+ VocabParallelEmbedding,+ get_rope,+)+from sglang_omni.vendor.sglang.models import apply_qk_norm+from sglang_omni.vendor.sglang.utils import make_layers++logger = logging.getLogger(__name__)+++ "audio8::fast_attention_with_cache",+ mutates_args=("key_cache", "value_cache"),+)+def _audio8_fast_attention_with_cache(+ query: Tensor,+ key_cache: Tensor,+ value_cache: Tensor,+ key: Tensor,+ value: Tensor,+ cache_positions: Tensor,+) -> Tensor:+ return flash_attn_with_kvcache(+ q=query,+ k_cache=key_cache,+ v_cache=value_cache,+ k=key,+ v=value,+ cache_seqlens=cache_positions.contiguous(),+ causal=True,+ num_splits=0,+ )+++@_audio8_fast_attention_with_cache.register_fake+def _audio8_fast_attention_with_cache_fake(+ query: Tensor,+ key_cache: Tensor,+ value_cache: Tensor,+ key: Tensor,+ value: Tensor,+ cache_positions: Tensor,+) -> Tensor:+ del key_cache, value_cache, key, value, cache_positions+ return torch.empty_like(query)+++class Audio8FastKVCache(nn.Module):+ def __init__(+ self,+ max_batch_size: int,+ max_sequence_length: int,+ num_kv_heads: int,+ head_dim: int,+ *,+ device: torch.device,+ dtype: torch.dtype,+ ) -> None:+ super().__init__()+ shape = (+ max_batch_size,+ max_sequence_length,+ num_kv_heads,+ head_dim,+ )+ self.register_buffer(+ "key",+ torch.zeros(shape, device=device, dtype=dtype),+ persistent=False,+ )+ self.register_buffer(+ "value",+ torch.zeros(shape, device=device, dtype=dtype),+ persistent=False,+ )++ def for_batch(self, batch_size: int) -> tuple[Tensor, Tensor]:+ return self.key[:batch_size], self.value[:batch_size]++ def clear(self) -> None:+ self.key.zero_()+ self.value.zero_()+++class Audio8Attention(nn.Module):+ def __init__(+ self,+ *,+ hidden_size: int,+ num_heads: int,+ num_kv_heads: int,+ head_dim: int,+ layer_id: int,+ rope_base: float,+ max_position_embeddings: int,+ rms_norm_eps: float,+ qkv_bias: bool,+ output_bias: bool,+ qk_norm: bool,+ ) -> None:+ super().__init__()+ self.q_size = num_heads * head_dim+ self.kv_size = num_kv_heads * head_dim+ self.head_dim = head_dim+ self.qkv_proj = QKVParallelLinear(+ hidden_size,+ head_dim,+ num_heads,+ num_kv_heads,+ bias=qkv_bias,+ )+ self.o_proj = RowParallelLinear(+ num_heads * head_dim,+ hidden_size,+ bias=output_bias,+ )+ self.rotary_emb = get_rope(+ head_dim,+ rotary_dim=head_dim,+ max_position=max_position_embeddings,+ base=rope_base,+ is_neox_style=False,+ )+ self.attn = RadixAttention(+ num_heads,+ head_dim,+ head_dim**-0.5,+ num_kv_heads=num_kv_heads,+ layer_id=layer_id,+ )+ self.qk_norm = qk_norm+ if qk_norm:+ self.q_norm = RMSNorm(head_dim, eps=rms_norm_eps)+ self.k_norm = RMSNorm(head_dim, eps=rms_norm_eps)++ def forward(+ self,+ positions: Tensor,+ hidden_states: Tensor,+ forward_batch: ForwardBatch,+ ) -> Tensor:+ qkv, _ = self.qkv_proj(hidden_states)+ q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)+ if self.qk_norm:+ q, k = apply_qk_norm(q, k, self.q_norm, self.k_norm, self.head_dim)+ q, k = self.rotary_emb(positions, q, k)+ output = self.attn(q, k, v, forward_batch)+ output, _ = self.o_proj(output)+ return output+++class Audio8DecoderLayer(nn.Module):+ def __init__(self, config: Any, layer_id: int) -> None:+ super().__init__()+ self.self_attn = Audio8Attention(+ hidden_size=config.dim,+ num_heads=config.n_head,+ num_kv_heads=config.n_local_heads,+ head_dim=config.head_dim,+ layer_id=layer_id,+ rope_base=config.rope_base,+ max_position_embeddings=config.max_seq_len,+ rms_norm_eps=config.norm_eps,+ qkv_bias=config.attention_qkv_bias,+ output_bias=config.attention_o_bias,+ qk_norm=config.attention_qk_norm,+ )+ self.gate_up_proj = MergedColumnParallelLinear(+ config.dim,+ [config.intermediate_size, config.intermediate_size],+ bias=False,+ )+ self.down_proj = RowParallelLinear(+ config.intermediate_size,+ config.dim,+ bias=False,+ )+ self.input_layernorm = RMSNorm(config.dim, eps=config.norm_eps)+ self.post_attention_layernorm = RMSNorm(config.dim, eps=config.norm_eps)++ def forward(+ self,+ positions: Tensor,+ hidden_states: Tensor,+ forward_batch: ForwardBatch,+ residual: Optional[Tensor],+ ) -> Tuple[Tensor, Tensor]:+ if residual is None:+ residual = hidden_states+ hidden_states = self.input_layernorm(hidden_states)+ else:+ hidden_states, residual = self.input_layernorm(hidden_states, residual)+ hidden_states = self.self_attn(positions, hidden_states, forward_batch)+ hidden_states, residual = self.post_attention_layernorm(hidden_states, residual)+ gate_up, _ = self.gate_up_proj(hidden_states)+ gate, up = gate_up.chunk(2, dim=-1)+ hidden_states = F.silu(gate) * up+ hidden_states, _ = self.down_proj(hidden_states)+ return hidden_states, residual+++class FastRMSNorm(nn.Module):+ def __init__(self, dim: int, eps: float) -> None:+ super().__init__()+ self.eps = float(eps)+ self.weight = nn.Parameter(torch.ones(dim))++ def forward(self, value: Tensor) -> Tensor:+ normalized = value.float() * torch.rsqrt(+ value.float().pow(2).mean(-1, keepdim=True) + self.eps+ )+ return normalized.to(value.dtype) * self.weight+++def _rope(length: int, head_dim: int, base: float, device: torch.device) -> Tensor:+ frequencies = 1.0 / (+ base ** (torch.arange(0, head_dim, 2, device=device).float() / head_dim)+ )+ phases = torch.outer(torch.arange(length, device=device), frequencies)+ values = torch.polar(torch.ones_like(phases), phases)+ return torch.stack((values.real, values.imag), dim=-1).to(torch.bfloat16)+++def _apply_rope(value: Tensor, rope_values: Tensor) -> Tensor:+ shaped = value.float().reshape(*value.shape[:-1], -1, 2)+ rope_values = rope_values[None, :, None]+ output = torch.stack(+ (+ shaped[..., 0] * rope_values[..., 0]+ - shaped[..., 1] * rope_values[..., 1],+ shaped[..., 1] * rope_values[..., 0]+ + shaped[..., 0] * rope_values[..., 1],+ ),+ dim=-1,+ )+ return output.flatten(3).to(value.dtype)+++class FastAttention(nn.Module):+ def __init__(self, config: Any) -> None:+ super().__init__()+ total = (config.fast_n_head + 2 * config.fast_n_local_heads) * config.fast_head_dim+ self.wqkv = nn.Linear(+ config.fast_dim,+ total,+ bias=config.fast_attention_qkv_bias,+ )+ self.wo = nn.Linear(+ config.fast_n_head * config.fast_head_dim,+ config.fast_dim,+ bias=config.fast_attention_o_bias,+ )+ self.n_head = config.fast_n_head+ self.n_local_heads = config.fast_n_local_heads+ self.head_dim = config.fast_head_dim+ self.qk_norm = config.fast_attention_qk_norm+ self.audio8_cache: Audio8FastKVCache | None = None+ if self.qk_norm:+ self.q_norm = FastRMSNorm(self.head_dim, config.norm_eps)+ self.k_norm = FastRMSNorm(self.head_dim, config.norm_eps)++ def setup_audio8_cache(+ self,+ max_batch_size: int,+ max_sequence_length: int,+ *,+ device: torch.device,+ dtype: torch.dtype,+ ) -> None:+ self.audio8_cache = Audio8FastKVCache(+ max_batch_size,+ max_sequence_length,+ self.n_local_heads,+ self.head_dim,+ device=device,+ dtype=dtype,+ )++ def clear_audio8_cache(self) -> None:+ if self.audio8_cache is None:+ raise RuntimeError("Audio8 fast KV cache has not been initialized")+ self.audio8_cache.clear()++ def forward(self, value: Tensor, rope_values: Tensor) -> Tensor:+ batch, length, _ = value.shape+ q_size = self.n_head * self.head_dim+ kv_size = self.n_local_heads * self.head_dim+ q, k, v = self.wqkv(value).split((q_size, kv_size, kv_size), dim=-1)+ q = q.view(batch, length, self.n_head, self.head_dim)+ k = k.view(batch, length, self.n_local_heads, self.head_dim)+ v = v.view(batch, length, self.n_local_heads, self.head_dim)+ if self.qk_norm:+ q = self.q_norm(q)+ k = self.k_norm(k)+ q = _apply_rope(q, rope_values).transpose(1, 2)+ k = _apply_rope(k, rope_values).transpose(1, 2)+ v = v.transpose(1, 2)+ repeat = self.n_head // self.n_local_heads+ k = k.repeat_interleave(repeat, dim=1)+ v = v.repeat_interleave(repeat, dim=1)+ scores = q @ k.transpose(-2, -1) / (self.head_dim**0.5)+ causal = torch.ones(+ length,+ length,+ dtype=torch.bool,+ device=value.device,+ ).tril()+ scores = scores.masked_fill(~causal[None, None], -float("inf"))+ output = torch.softmax(scores, dim=-1) @ v+ return self.wo(output.transpose(1, 2).contiguous().view(batch, length, q_size))++ def forward_audio8_cached(+ self,+ value: Tensor,+ rope_values: Tensor,+ cache_positions: Tensor,+ ) -> Tensor:+ batch, length, _ = value.shape+ if length != 1:+ raise ValueError("Audio8 fast KV-cache decode expects one token per step")+ q_size = self.n_head * self.head_dim+ kv_size = self.n_local_heads * self.head_dim+ q, k, v = self.wqkv(value).split((q_size, kv_size, kv_size), dim=-1)+ q = q.view(batch, length, self.n_head, self.head_dim)+ k = k.view(batch, length, self.n_local_heads, self.head_dim)+ v = v.view(batch, length, self.n_local_heads, self.head_dim)+ if self.qk_norm:+ q = self.q_norm(q)+ k = self.k_norm(k)+ q = _apply_rope(q, rope_values)+ k = _apply_rope(k, rope_values)+ if self.audio8_cache is None:+ raise RuntimeError("Audio8 fast KV cache has not been initialized")+ key_cache, value_cache = self.audio8_cache.for_batch(batch)+ output = _audio8_fast_attention_with_cache(+ q,+ key_cache,+ value_cache,+ k,+ v,+ cache_positions,+ )+ return self.wo(output.contiguous().view(batch, length, q_size))+++class FastFeedForward(nn.Module):+ def __init__(self, config: Any) -> None:+ super().__init__()+ self.w1 = nn.Linear(config.fast_dim, config.fast_intermediate_size, bias=False)+ self.w2 = nn.Linear(config.fast_intermediate_size, config.fast_dim, bias=False)+ self.w3 = nn.Linear(config.fast_dim, config.fast_intermediate_size, bias=False)++ def forward(self, value: Tensor) -> Tensor:+ return self.w2(F.silu(self.w1(value)) * self.w3(value))+++class FastDecoderLayer(nn.Module):+ def __init__(self, config: Any) -> None:+ super().__init__()+ self.attention = FastAttention(config)+ self.feed_forward = FastFeedForward(config)+ self.ffn_norm = FastRMSNorm(config.fast_dim, config.norm_eps)+ self.attention_norm = FastRMSNorm(config.fast_dim, config.norm_eps)++ def forward(self, value: Tensor, rope_values: Tensor) -> Tensor:+ hidden = value + self.attention(self.attention_norm(value), rope_values)+ return hidden + self.feed_forward(self.ffn_norm(hidden))++ def forward_audio8_cached(+ self,+ value: Tensor,+ rope_values: Tensor,+ cache_positions: Tensor,+ ) -> Tensor:+ hidden = value + self.attention.forward_audio8_cached(+ self.attention_norm(value),+ rope_values,+ cache_positions,+ )+ return hidden + self.feed_forward(self.ffn_norm(hidden))+++class Audio8SGLangModel(nn.Module):+ def __init__(self, config: Any, quant_config: Any = None) -> None:+ super().__init__()+ del quant_config+ self.config = config+ self.vocab_size = int(config.vocab_size)+ self.hidden_size = int(config.dim)+ self.num_layers = int(config.n_layer)+ self.tie_word_embeddings = bool(config.tie_word_embeddings)+ self.embed_tokens = VocabParallelEmbedding(config.vocab_size, config.dim)+ self.codebook_embeddings = nn.Embedding(+ config.codebook_size * config.num_codebooks,+ config.dim,+ )+ self.start_layer = 0+ self.end_layer = config.n_layer+ self.layers = make_layers(+ config.n_layer,+ lambda idx, prefix: Audio8DecoderLayer(config, idx),+ prefix="layers",+ )+ self.norm = RMSNorm(config.dim, eps=config.norm_eps)++ self.fast_project_in = (+ nn.Linear(config.dim, config.fast_dim)+ if config.fast_dim != config.dim+ else nn.Identity()+ )+ self.fast_embeddings = nn.Embedding(config.codebook_size, config.fast_dim)+ self.fast_layers = nn.ModuleList(+ [FastDecoderLayer(config) for _ in range(config.n_fast_layer)]+ )+ self.fast_norm = FastRMSNorm(config.fast_dim, config.norm_eps)+ self.fast_output = nn.Linear(config.fast_dim, config.codebook_size, bias=False)+ self._decode_ready = False++ def setup_audio8_decode(self, max_batch_size: int) -> None:+ device = self.embed_tokens.weight.device+ config = self.config+ self.register_buffer(+ "_codebook_offsets",+ torch.arange(config.num_codebooks, device=device, dtype=torch.long)+ * config.codebook_size,+ persistent=False,+ )+ self.register_buffer(+ "_vq_codes",+ torch.zeros(max_batch_size, config.num_codebooks, device=device, dtype=torch.long),+ persistent=False,+ )+ self.register_buffer(+ "_vq_mask",+ torch.zeros(max_batch_size, device=device, dtype=torch.bool),+ persistent=False,+ )+ semantic_bias = torch.full(+ (config.vocab_size,),+ -float("inf"),+ device=device,+ dtype=torch.float32,+ )+ semantic_bias[config.semantic_begin_id : config.semantic_end_id + 1] = 0.0+ semantic_bias[config.eos_token_id] = 0.0+ self.register_buffer("_semantic_bias", semantic_bias, persistent=False)+ self.register_buffer(+ "_output_codes",+ torch.zeros(+ max_batch_size,+ config.num_codebooks + 1,+ device=device,+ dtype=torch.long,+ ),+ persistent=False,+ )+ self.register_buffer(+ "_output_semantic_ids",+ torch.zeros(max_batch_size, device=device, dtype=torch.long),+ persistent=False,+ )+ self.register_buffer(+ "_temperature",+ torch.full((max_batch_size,), 0.8, device=device),+ persistent=False,+ )+ self.register_buffer(+ "_top_p",+ torch.full((max_batch_size,), 0.95, device=device),+ persistent=False,+ )+ self.register_buffer(+ "_top_k",+ torch.full((max_batch_size,), 50, device=device, dtype=torch.long),+ persistent=False,+ )+ self.register_buffer(+ "_do_sample",+ torch.ones(max_batch_size, device=device, dtype=torch.bool),+ persistent=False,+ )+ self.register_buffer(+ "_previous_semantic",+ torch.zeros(+ max_batch_size,+ config.ras_window_size,+ device=device,+ dtype=torch.long,+ ),+ persistent=False,+ )+ self.register_buffer(+ "_previous_valid",+ torch.zeros(+ max_batch_size,+ config.ras_window_size,+ device=device,+ dtype=torch.bool,+ ),+ persistent=False,+ )+ self.register_buffer(+ "_fast_rope",+ _rope(+ config.num_codebooks,+ config.fast_head_dim,+ config.rope_base,+ device,+ ),+ persistent=False,+ )+ self.register_buffer(+ "_audio8_fast_position",+ torch.zeros(1, device=device, dtype=torch.long),+ persistent=False,+ )+ for layer in self.fast_layers:+ layer.attention.setup_audio8_cache(+ max_batch_size,+ config.num_codebooks + 1,+ device=device,+ dtype=self.embed_tokens.weight.dtype,+ )+ self._decode_ready = True++ def _embed_decode(self, input_ids: Tensor) -> Tensor:+ hidden = self.embed_tokens(input_ids)+ batch = hidden.shape[0]+ codes = self._vq_codes[:batch] + self._codebook_offsets[None]+ codebook_sum = self.codebook_embeddings(codes).sum(dim=1).to(hidden.dtype)+ return torch.where(+ self._vq_mask[:batch, None],+ hidden + codebook_sum,+ hidden,+ )++ @staticmethod+ def _sample(+ scores: Tensor,+ temperature: Tensor,+ top_p: Tensor,+ top_k: Tensor,+ do_sample: Tensor,+ ) -> Tensor:+ sorted_scores, sorted_indices = torch.sort(scores, descending=True, dim=-1)+ cumulative = torch.softmax(sorted_scores, dim=-1).cumsum(dim=-1)+ positions = torch.arange(scores.shape[-1], device=scores.device)[None]+ remove_sorted = (cumulative > top_p[:, None]) | (positions >= top_k[:, None])+ remove_sorted[:, 0] = False+ remove = torch.zeros_like(remove_sorted).scatter(1, sorted_indices, remove_sorted)+ filtered = scores.masked_fill(remove, -float("inf"))+ filtered = filtered / temperature[:, None].clamp_min(1e-5)+ greedy = filtered.argmax(dim=-1)+ probabilities = torch.softmax(filtered, dim=-1)+ random = torch.rand_like(probabilities).clamp_min(torch.finfo(probabilities.dtype).tiny)+ sampled = torch.argmax(probabilities / (-torch.log(random)), dim=-1)+ return torch.where(do_sample, sampled, greedy)++ def _sample_semantic(self, logits: Tensor) -> Tensor:+ batch = logits.shape[0]+ scores = logits.float() + self._semantic_bias+ normal = self._sample(+ scores,+ self._temperature[:batch],+ self._top_p[:batch],+ self._top_k[:batch],+ self._do_sample[:batch],+ )+ ras_temperature = torch.full_like(+ self._temperature[:batch], self.config.ras_temperature+ )+ ras_top_p = torch.full_like(self._top_p[:batch], self.config.ras_top_p)+ high = self._sample(+ scores,+ ras_temperature,+ ras_top_p,+ self._top_k[:batch],+ self._do_sample[:batch],+ )+ repeated = (+ (self._previous_semantic[:batch] == normal[:, None])+ & self._previous_valid[:batch]+ ).any(dim=1)+ is_semantic = (normal >= self.config.semantic_begin_id) & (+ normal <= self.config.semantic_end_id+ )+ return torch.where(repeated & is_semantic, high, normal)++ def _clear_audio8_fast_caches(self) -> None:+ for layer in self.fast_layers:+ layer.attention.clear_audio8_cache()++ def _run_audio8_fast_position(self, hidden: Tensor, position: int) -> Tensor:+ batch = hidden.shape[0]+ self._audio8_fast_position.fill_(position)+ rope_values = self._fast_rope[self._audio8_fast_position]+ cache_positions = self._audio8_fast_position.expand(batch).to(torch.int32)+ for layer in self.fast_layers:+ hidden = layer.forward_audio8_cached(+ hidden,+ rope_values,+ cache_positions,+ )+ return self.fast_output(self.fast_norm(hidden))[:, -1]++ @torch.no_grad()+ def _decode_codebooks(self, logits: Tensor, slow_hidden: Tensor) -> None:+ batch = logits.shape[0]+ semantic = self._sample_semantic(logits)+ current = (semantic - self.config.semantic_begin_id).clamp(+ 0, self.config.codebook_size - 1+ )+ self._output_codes[:batch, 0] = semantic+ self._output_codes[:batch, 1] = current+ self._clear_audio8_fast_caches()+ fast_hidden = self.fast_project_in(slow_hidden).unsqueeze(1)+ self._run_audio8_fast_position(fast_hidden, 0)+ for position in range(1, self.config.num_codebooks):+ fast_hidden = self.fast_embeddings(current).unsqueeze(1)+ scores = self._run_audio8_fast_position(fast_hidden, position).float()+ current = self._sample(+ scores,+ self._temperature[:batch],+ self._top_p[:batch],+ self._top_k[:batch],+ self._do_sample[:batch],+ )+ self._output_codes[:batch, position + 1] = current+ self._output_semantic_ids[:batch] = semantic++ def forward(+ self,+ input_ids: Tensor,+ positions: Tensor,+ forward_batch: ForwardBatch,+ input_embeds: Optional[Tensor] = None,+ ) -> LogitsProcessorOutput:+ if input_embeds is None and forward_batch.input_embeds is not None:+ input_embeds = forward_batch.input_embeds+ if input_embeds is not None:+ hidden_states = input_embeds+ elif self._decode_ready:+ hidden_states = self._embed_decode(input_ids)+ else:+ hidden_states = self.embed_tokens(input_ids)++ residual = None+ for layer_index in range(self.start_layer, self.end_layer):+ hidden_states, residual = self.layers[layer_index](+ positions, hidden_states, forward_batch, residual+ )+ hidden_states, _ = self.norm(hidden_states, residual)+ if forward_batch.forward_mode.is_extend():+ last_index = torch.cumsum(forward_batch.extend_seq_lens, dim=0) - 1+ hidden_states = hidden_states[last_index]+ logits = F.linear(hidden_states, self.embed_tokens.weight)+ if self._decode_ready:+ self._decode_codebooks(logits, hidden_states)+ return LogitsProcessorOutput(+ next_token_logits=logits,+ hidden_states=hidden_states,+ )++ def get_embed_tokens(self) -> nn.Module:+ return self.embed_tokens++ def load_weights(self, weights: Iterable[Tuple[str, Tensor]]) -> None:+ params = dict(self.named_parameters())+ for name, loaded_weight in weights:+ if name in {"freqs_cis", "fast_freqs_cis"}:+ continue+ if self._load_slow_weight(name, loaded_weight, params):+ continue+ target_name = "embed_tokens.weight" if name == "embeddings.weight" else name+ target = params.get(target_name)+ if target is None:+ logger.debug("Skipping Audio8 weight: %s", name)+ continue+ loader = getattr(target, "weight_loader", _default_weight_loader)+ loader(target, loaded_weight)++ def _load_slow_weight(+ self,+ name: str,+ loaded_weight: Tensor,+ params: dict[str, nn.Parameter],+ ) -> bool:+ if not name.startswith("layers."):+ return False+ remap: dict[str, Any] = {+ "attention.wqkv.weight": None,+ "attention.wqkv.bias": None,+ "attention.wo.weight": "self_attn.o_proj.weight",+ "attention.wo.bias": "self_attn.o_proj.bias",+ "attention.q_norm.weight": "self_attn.q_norm.weight",+ "attention.k_norm.weight": "self_attn.k_norm.weight",+ "attention_norm.weight": "input_layernorm.weight",+ "ffn_norm.weight": "post_attention_layernorm.weight",+ "feed_forward.w1.weight": ("gate_up_proj.weight", 0),+ "feed_forward.w3.weight": ("gate_up_proj.weight", 1),+ "feed_forward.w2.weight": "down_proj.weight",+ }+ for source_suffix, target in remap.items():+ if not name.endswith(source_suffix):+ continue+ prefix = name[: -len(source_suffix)]+ if target is None:+ self._load_fused_qkv(+ prefix,+ loaded_weight,+ params,+ is_bias=source_suffix.endswith("bias"),+ )+ return True+ if isinstance(target, tuple):+ target_suffix, shard_id = target+ else:+ target_suffix, shard_id = target, None+ parameter = params[prefix + target_suffix]+ if shard_id is None:+ loader = getattr(parameter, "weight_loader", _default_weight_loader)+ loader(parameter, loaded_weight)+ else:+ parameter.weight_loader(parameter, loaded_weight, shard_id)+ return True+ return False++ def _load_fused_qkv(+ self,+ prefix: str,+ loaded_weight: Tensor,+ params: dict[str, nn.Parameter],+ *,+ is_bias: bool,+ ) -> None:+ target_name = prefix + "self_attn.qkv_proj." + ("bias" if is_bias else "weight")+ parameter = params[target_name]+ layer = self.layers[int(prefix.split(".")[1])]+ q, k, v = loaded_weight.split(+ [layer.self_attn.q_size, layer.self_attn.kv_size, layer.self_attn.kv_size],+ dim=0,+ )+ for shard_id, weight in (("q", q), ("k", k), ("v", v)):+ parameter.weight_loader(parameter, weight, shard_id)+++def _default_weight_loader(parameter: nn.Parameter, loaded_weight: Tensor) -> None:+ parameter.data.copy_(loaded_weight)+++EntryClass = Audio8SGLangModeldiff --git a/sglang_omni/adapter/sglang_omni/models/audio8_tts/tokenizer.py b/sglang_omni/adapter/sglang_omni/models/audio8_tts/tokenizer.pynew file mode 100644index 0000000..be786ab--- /dev/null+++ b/sglang_omni/adapter/sglang_omni/models/audio8_tts/tokenizer.py@@ -0,0 +1,114 @@+# SPDX-License-Identifier: Apache-2.0+from __future__ import annotations++import inspect+import re+from dataclasses import dataclass+from typing import Any++import torch+++@dataclass+class Reference:+ text: str+ vq_codes: torch.Tensor | None = None+++class Audio8TokenizerAdapter:+ def __init__(+ self,+ tokenizer: Any,+ *,+ semantic_begin_id: int,+ semantic_end_id: int,+ eos_token_id: int,+ ) -> None:+ self._tok = tokenizer+ self.semantic_begin_id = int(semantic_begin_id)+ self.semantic_end_id = int(semantic_end_id)+ self.eos_token_ids = [int(eos_token_id)]++ def _encode(self, text: str) -> torch.Tensor:+ kwargs: dict[str, Any] = {"add_special_tokens": False}+ if "allowed_special" in inspect.signature(self._tok.encode).parameters:+ kwargs["allowed_special"] = "all"+ return torch.tensor(self._tok.encode(text, **kwargs), dtype=torch.long)++ @staticmethod+ def _clean(text: str) -> str:+ return " ".join(str(text).strip().split())++ @classmethod+ def _reference_text(cls, text: str) -> str:+ text = cls._clean(text)+ return text if re.search(r"<\|speaker:\d+\|>", text) else f"<|speaker:0|>{text}"++ def build_prompt(+ self,+ text: str,+ references: list[Reference] | None = None,+ *,+ num_codebooks: int = 10,+ **_: Any,+ ) -> dict[str, Any]:+ target = self._clean(text)+ if not target:+ raise ValueError("text must not be empty")++ def encode_parts(parts: list[str]) -> torch.Tensor:+ return torch.cat([self._encode(part) for part in parts])++ if not references:+ tokens = encode_parts(+ [+ "<|im_start|>system\n",+ "convert the provided text to speech",+ "<|im_end|>\n",+ "<|im_start|>user\n",+ target,+ "<|im_end|>\n",+ "<|im_start|>assistant\n<|voice|>",+ ]+ )+ return {+ "input_ids": tokens,+ "vq_mask_tokens": torch.zeros(tokens.numel(), dtype=torch.bool),+ "vq_parts": [],+ }++ if len(references) != 1:+ raise ValueError("Audio8 TTS currently supports exactly one reference")+ reference = references[0]+ if not reference.text:+ raise ValueError("reference text is required")+ if reference.vq_codes is None:+ raise ValueError("reference VQ codes are required")+ codes = reference.vq_codes.to(dtype=torch.long, device="cpu")+ if codes.ndim != 2 or codes.shape[0] != num_codebooks or codes.shape[1] == 0:+ raise ValueError(+ f"reference codes must have shape [{num_codebooks}, T>0], got {tuple(codes.shape)}"+ )++ prefix = encode_parts(+ [+ "<|im_start|>system\n",+ "convert the provided text to speech reference to the following:\n\nText:\n",+ self._reference_text(reference.text),+ "\n\nSpeech:\n",+ ]+ )+ semantic = codes[0] + self.semantic_begin_id+ suffix = encode_parts(+ [+ "<|im_end|>\n",+ "<|im_start|>user\n",+ target,+ "<|im_end|>\n",+ "<|im_start|>assistant\n<|voice|>",+ ]+ )+ tokens = torch.cat((prefix, semantic, suffix))+ mask = torch.zeros(tokens.numel(), dtype=torch.bool)+ mask[prefix.numel() : prefix.numel() + semantic.numel()] = True+ return {"input_ids": tokens, "vq_mask_tokens": mask, "vq_parts": [codes]}diff --git a/sglang_omni/configs/audio8_tts_0_6b.yaml b/sglang_omni/configs/audio8_tts_0_6b.yamlnew file mode 100644index 0000000..6d67b19--- /dev/null+++ b/sglang_omni/configs/audio8_tts_0_6b.yaml@@ -0,0 +1,35 @@+# SPDX-License-Identifier: Apache-2.0+config_cls: Audio8TTSPipelineConfig+name: audio8-tts-0.6b+model_path: /models/Audio8-TTS-Preview-0.6b+relay_backend: shm+entry_stage: preprocessing+stages:+ - name: preprocessing+ num_workers: 1+ executor:+ factory: sglang_omni.models.audio8_tts.pipeline.stages.create_preprocessing_executor+ args:+ device: cuda:0+ get_next: sglang_omni.models.audio8_tts.pipeline.next_stage.preprocessing_next+ relay:+ device: cpu+ - name: tts_engine+ num_workers: 1+ executor:+ factory: sglang_omni.models.audio8_tts.pipeline.stages.create_sglang_tts_engine_executor+ args:+ device: cuda:0+ max_new_tokens: 1024+ get_next: sglang_omni.models.audio8_tts.pipeline.next_stage.tts_engine_next+ relay:+ device: cuda+ - name: vocoder+ num_workers: 1+ executor:+ factory: sglang_omni.models.audio8_tts.pipeline.stages.create_vocoder_executor+ args:+ device: cuda:0+ get_next: sglang_omni.models.audio8_tts.pipeline.next_stage.vocoder_next+ relay:+ device: cpudiff --git a/sglang_omni/scripts/install_adapter.sh b/sglang_omni/scripts/install_adapter.shnew file mode 100755index 0000000..2a77503--- /dev/null+++ b/sglang_omni/scripts/install_adapter.sh@@ -0,0 +1,28 @@+#!/usr/bin/env bash+# SPDX-License-Identifier: Apache-2.0+set -euo pipefail++if [[ $# -ne 1 ]]; then+ echo "Usage: $0 <sglang-omni source root or sglang_omni package directory>" >&2+ exit 2+fi++BUNDLE_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"+SOURCE_DIR="${BUNDLE_ROOT}/adapter/sglang_omni/models/audio8_tts"+TARGET="$1"++if [[ -d "${TARGET}/sglang_omni/models" ]]; then+ PACKAGE_DIR="${TARGET}/sglang_omni"+elif [[ -d "${TARGET}/models" && -f "${TARGET}/__init__.py" ]]; then+ PACKAGE_DIR="${TARGET}"+else+ echo "Cannot find sglang_omni/models under: ${TARGET}" >&2+ exit 1+fi++DEST_DIR="${PACKAGE_DIR}/models/audio8_tts"+mkdir -p "${DEST_DIR}"+cp -a "${SOURCE_DIR}/." "${DEST_DIR}/"++echo "Installed Audio8 adapter to: ${DEST_DIR}"+echo "No SGLang-Omni core files were modified."diff --git a/sglang_omni/scripts/run_server.sh b/sglang_omni/scripts/run_server.shnew file mode 100755index 0000000..66b742b--- /dev/null+++ b/sglang_omni/scripts/run_server.sh@@ -0,0 +1,40 @@+#!/usr/bin/env bash+# SPDX-License-Identifier: Apache-2.0+set -euo pipefail++BUNDLE_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"+PYTHON_BIN="${PYTHON_BIN:-python3}"+MODEL="${MODEL:-/models/Audio8-TTS-Preview-0.6b}"+CONFIG="${CONFIG:-${BUNDLE_ROOT}/configs/audio8_tts_0_6b.yaml}"+MODEL_NAME="${MODEL_NAME:-audio8/tts-0.6b}"+HOST="${HOST:-0.0.0.0}"+PORT="${PORT:-8010}"++if [[ -n "${SGLANG_OMNI_ROOT:-}" ]]; then+ if [[ -d "${SGLANG_OMNI_ROOT}/sglang_omni" ]]; then+ export PYTHONPATH="${SGLANG_OMNI_ROOT}${PYTHONPATH:+:${PYTHONPATH}}"+ elif [[ -f "${SGLANG_OMNI_ROOT}/__init__.py" ]]; then+ PACKAGE_PARENT="$(cd "${SGLANG_OMNI_ROOT}/.." && pwd)"+ export PYTHONPATH="${PACKAGE_PARENT}${PYTHONPATH:+:${PYTHONPATH}}"+ else+ echo "Invalid SGLANG_OMNI_ROOT: ${SGLANG_OMNI_ROOT}" >&2+ exit 1+ fi+fi++if [[ -n "${SGLANG_OMNI_SITE_PACKAGES:-}" ]]; then+ if [[ ! -d "${SGLANG_OMNI_SITE_PACKAGES}" ]]; then+ echo "Invalid SGLANG_OMNI_SITE_PACKAGES: ${SGLANG_OMNI_SITE_PACKAGES}" >&2+ exit 1+ fi+ export PYTHONPATH="${PYTHONPATH:-}${PYTHONPATH:+:}${SGLANG_OMNI_SITE_PACKAGES}"+fi++export FLASHINFER_WORKSPACE_BASE="${FLASHINFER_WORKSPACE_BASE:-/tmp/audio8-flashinfer}"++exec "${PYTHON_BIN}" -m sglang_omni.cli.cli serve \+ --model-path "${MODEL}" \+ --config "${CONFIG}" \+ --model-name "${MODEL_NAME}" \+ --host "${HOST}" \+ --port "${PORT}"diff --git a/sglang_omni/scripts/smoke_test.sh b/sglang_omni/scripts/smoke_test.shnew file mode 100755index 0000000..ed8c2b6--- /dev/null+++ b/sglang_omni/scripts/smoke_test.sh@@ -0,0 +1,27 @@+#!/usr/bin/env bash+# SPDX-License-Identifier: Apache-2.0+set -euo pipefail++BASE_URL="${BASE_URL:-http://127.0.0.1:8010}"+OUTPUT="${OUTPUT:-/tmp/audio8_sglang_omni_smoke.wav}"++curl -sS \+ -H 'Content-Type: application/json' \+ -d '{+ "model": "audio8/tts-0.6b",+ "input": "你好,这是 Audio8 TTS 的 SGLang Omni 推理测试。",+ "response_format": "wav",+ "max_new_tokens": 128,+ "temperature": 0.8,+ "top_p": 0.95,+ "top_k": 50+ }' \+ "${BASE_URL}/v1/audio/speech" \+ -o "${OUTPUT}"++magic="$(od -An -c -N4 "${OUTPUT}" | tr -d ' ')"+if [[ "${magic}" != "RIFF" ]]; then+ echo "Unexpected response header: ${magic}" >&2+ exit 1+fi+wc -c "${OUTPUT}"diff --git a/sglang_omni/scripts/test_fast_kv_cache.py b/sglang_omni/scripts/test_fast_kv_cache.pynew file mode 100755index 0000000..91ba259--- /dev/null+++ b/sglang_omni/scripts/test_fast_kv_cache.py@@ -0,0 +1,99 @@+#!/usr/bin/env python3+# SPDX-License-Identifier: Apache-2.0+from __future__ import annotations++import argparse+import json+from pathlib import Path+from types import SimpleNamespace++import torch++from sglang_omni.models.audio8_tts.sglang_model import FastDecoderLayer, _rope+++def main() -> None:+ parser = argparse.ArgumentParser()+ parser.add_argument("--model-path", type=Path, required=True)+ parser.add_argument("--batch-size", type=int, default=2)+ parser.add_argument("--seed", type=int, default=1234)+ args = parser.parse_args()++ config = SimpleNamespace(+ **json.loads((args.model_path / "config.json").read_text(encoding="utf-8"))+ )+ if not torch.cuda.is_available():+ raise SystemExit("CUDA is required for the Fast AR KV-cache test")++ torch.manual_seed(args.seed)+ torch.cuda.manual_seed_all(args.seed)+ device = torch.device("cuda")+ layers = torch.nn.ModuleList(+ [FastDecoderLayer(config) for _ in range(config.n_fast_layer)]+ ).to(device=device, dtype=torch.bfloat16).eval()+ for layer in layers:+ layer.attention.setup_audio8_cache(+ args.batch_size,+ config.num_codebooks + 1,+ device=device,+ dtype=torch.bfloat16,+ )++ inputs = torch.randn(+ args.batch_size,+ config.num_codebooks,+ config.fast_dim,+ device=device,+ dtype=torch.bfloat16,+ )+ rope = _rope(+ config.num_codebooks,+ config.fast_head_dim,+ config.rope_base,+ device,+ )++ with torch.inference_mode():+ full = inputs+ for layer in layers:+ full = layer(full, rope)++ for layer in layers:+ layer.attention.clear_audio8_cache()+ cached_parts = []+ for position in range(config.num_codebooks):+ hidden = inputs[:, position : position + 1]+ cache_positions = torch.full(+ (args.batch_size,),+ position,+ device=device,+ dtype=torch.int32,+ )+ for layer in layers:+ hidden = layer.forward_audio8_cached(+ hidden,+ rope[position : position + 1],+ cache_positions,+ )+ cached_parts.append(hidden)+ cached = torch.cat(cached_parts, dim=1)++ difference = (full.float() - cached.float()).abs()+ max_abs = float(difference.max().item())+ mean_abs = float(difference.mean().item())+ cosine = float(+ torch.nn.functional.cosine_similarity(+ full.float().flatten(),+ cached.float().flatten(),+ dim=0,+ ).item()+ )+ print(f"max_abs={max_abs:.6f}")+ print(f"mean_abs={mean_abs:.6f}")+ print(f"cosine={cosine:.8f}")+ if cosine < 0.999:+ raise SystemExit("Fast KV-cache output diverged from full causal forward")+++if __name__ == "__main__":+ main()diff --git a/sglang_omni/scripts/verify_install.py b/sglang_omni/scripts/verify_install.pynew file mode 100755index 0000000..9be74d0--- /dev/null+++ b/sglang_omni/scripts/verify_install.py@@ -0,0 +1,47 @@+#!/usr/bin/env python3+# SPDX-License-Identifier: Apache-2.0+from __future__ import annotations++import argparse+import json+from pathlib import Path+++def main() -> None:+ parser = argparse.ArgumentParser()+ parser.add_argument("--model-path", type=Path)+ args = parser.parse_args()++ from sglang_omni.models.audio8_tts.config import Audio8TTSPipelineConfig+ from sglang_omni.models.audio8_tts.factory import make_server_args+ from sglang_omni.models.audio8_tts.sglang_model import (+ Audio8SGLangModel,+ FastAttention,+ )++ assert Audio8TTSPipelineConfig.architecture == "ArkttsModel"+ assert Audio8SGLangModel.__name__ == "Audio8SGLangModel"+ assert callable(FastAttention.forward_audio8_cached)+ assert callable(make_server_args)++ if args.model_path is not None:+ required = {+ "config.json",+ "model.safetensors",+ "codec.pth",+ "tokenizer.json",+ "tokenizer_config.json",+ "modeling_arktts_codec.py",+ }+ missing = sorted(name for name in required if not (args.model_path / name).is_file())+ if missing:+ raise SystemExit(f"Model directory is missing: {missing}")+ config = json.loads((args.model_path / "config.json").read_text())+ if config.get("architectures") != ["ArkttsModel"]:+ raise SystemExit("config.json architectures must contain ArkttsModel")++ print("Audio8 SGLang-Omni adapter import check passed")+++if __name__ == "__main__":+ main()