SOTA-Class TTS at Compact Scale
Python
Add CPU-only ONNX INT4 runtime
linyubupa5f90e9fb82ce
diff --git a/.gitignore b/.gitignoreindex ce29964..6d4454a 100644--- a/.gitignore+++ b/.gitignore@@ -11,6 +11,11 @@ checkpoints/prepared_data/*.log+/onnx_runtime/model/+/onnx_runtime/voices/+/onnx_runtime/outputs/+/onnx_runtime/.service.pid+# Model weights are distributed separately from the source repository./model//configs/diff --git a/README.md b/README.mdindex 8e17f33..0fe1aae 100644--- a/README.md+++ b/README.md@@ -8,7 +8,8 @@[](https://github.com/Audio8-AI/Audio8_TTS)[](https://audio8-ai.github.io/Audio8_TTS/)-[](https://huggingface.co/AutoArk-AI/Audio8-TTS-Preview-0.6b)+[](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6b)+[](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4)[](LICENSE)中文文档: [README_zh.md](README_zh.md)@@ -73,7 +74,7 @@ pip install -r requirements.txt```Download the checkpoint from-[Hugging Face](https://huggingface.co/AutoArk-AI/Audio8-TTS-Preview-0.6b) and+[Hugging Face](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6b) andplace it in the repository's `model/` directory. The expected local checkpointpath is `model/audio8_tts_0_6B_preview/`. All commands also accept a HuggingFace model ID through `--model`.@@ -122,6 +123,21 @@ The batch command writes `manifest.jsonl` and `failures.jsonl`. Existing WAVfiles are skipped unless `--overwrite` is passed. See`python audio8_tts_infer.py --help` for sampling and code-saving options.+## CPU ONNX Runtime++[`onnx_runtime/`](onnx_runtime/) provides a standalone CPU deployment using+weight-only INT4 Slow/Fast AR models, FP16 activations and KV caches, and an+FP16 codec. It includes CLI inference, a local web and HTTP service, streaming+PCM output, and reference-voice registration without PyTorch or Transformers.++The online sessions use about 1 GiB of memory in the tested Apple M2 setup.+During voice registration, the online sessions are released before the codec+encoder is loaded to keep peak memory low.++Download the ONNX model from+[Audio8-TTS-Preview-0.6B-ONNX-INT4](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4)+and follow the [ONNX Runtime guide](onnx_runtime/README.md).+## SGLang Omni ServingThe adapter in [`sglang_omni/`](sglang_omni/) provides an OpenAI-compatiblediff --git a/README_zh.md b/README_zh.mdindex e356e3b..d43fdab 100644--- a/README_zh.md+++ b/README_zh.md@@ -8,7 +8,8 @@[](https://github.com/Audio8-AI/Audio8_TTS)[](https://audio8-ai.github.io/Audio8_TTS/)-[](https://huggingface.co/AutoArk-AI/Audio8-TTS-Preview-0.6b)+[](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6b)+[](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4)[](LICENSE)English: [README.md](README.md)@@ -112,6 +113,20 @@ python audio8_tts_infer.py \批量命令会生成 `manifest.jsonl` 和 `failures.jsonl`。除非传入 `--overwrite`,否则已有 WAV 会被跳过。采样和 codec codes 保存参数可通过 `python audio8_tts_infer.py --help` 查看。+## CPU ONNX Runtime++[`onnx_runtime/`](onnx_runtime/) 提供独立的 CPU 部署方案:Slow/Fast AR 使用+weight-only INT4,activation 和 KV cache 使用 FP16,codec 使用 FP16。它包含+命令行推理、本地网页与 HTTP 服务、流式 PCM 输出和参考音频音色注册,运行时不依赖+PyTorch 或 Transformers。++在 Apple M2 测试环境中,在线模型 session 约占 1 GiB 内存。注册音色前会先释放+在线 session,再单独加载 codec encoder,从而控制峰值内存。++从+[Audio8-TTS-Preview-0.6B-ONNX-INT4](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4)+下载模型,并参阅 [ONNX Runtime 中文指南](onnx_runtime/README_zh.md)。+## SGLang Omni 服务部署[`sglang_omni/`](sglang_omni/) 中的适配器提供兼容 OpenAI API 的推理服务,包含 SGLangdiff --git a/onnx_runtime/README.md b/onnx_runtime/README.mdnew file mode 100644index 0000000..f82dc30--- /dev/null+++ b/onnx_runtime/README.md@@ -0,0 +1,151 @@+# Audio8 TTS ONNX Runtime++CPU-only ONNX Runtime inference for+[Audio8-TTS-Preview-0.6B-ONNX-INT4](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4),+including CLI inference, a web and HTTP service, streaming PCM output, and+reference-voice registration. The runtime does not require PyTorch,+Transformers, or Hugging Face Hub after the model files have been downloaded.++Chinese documentation: [README_zh.md](README_zh.md)++## Precision and resource use++| Component | Precision |+|---|---|+| Slow/Fast AR weights | Weight-only INT4 |+| Activations, hidden states, and KV cache | FP16 |+| Codec encoder and decoder | FP16 |+| Waveform output | FP32 |++Normal synthesis loads only the Slow AR, Fast AR, and codec decoder sessions.+On a 16 GB Apple M2 MacBook Air with five ONNX Runtime threads, the service+used about 1004 MiB after loading and approximately 1.1-1.2 GiB at synthesis+peak. Voice registration unloads those sessions before loading the encoder;+the measured registration peak was approximately 1.55 GiB. Measurements vary+by operating system and ONNX Runtime allocator behavior.++The online model files occupy about 572 MiB. The optional voice-registration+encoder brings the complete download to about 968 MiB.++## Install++Python 3.11 or newer is required. The current release is tested on macOS arm64+with `CPUExecutionProvider`.++```bash+git clone https://github.com/Audio8-AI/Audio8_TTS.git+cd Audio8_TTS/onnx_runtime++python3 -m pip install -U "huggingface_hub[cli]"+hf download Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4 --local-dir model+bash setup.sh+```++The downloaded directory must have this layout:++```text+model/+|- slow_ar_int4.onnx(.data)+|- fast_ar_int4.onnx(.data)+|- codec_decoder_fp16.onnx(.data)+|- runtime_manifest.json+|- tokenizer/tokenizer.json+`- registration/+ |- codec_encoder_fp16.onnx(.data)+ `- registration_manifest.json+```++Set `ARKTTS_MODEL_DIR` if the model is stored elsewhere. Voice profiles are+stored in `voices/` by default; override this with `ARKTTS_VOICES_DIR`.++## Register a voice++A voice profile contains codec codes extracted from a reference recording and+its exact transcript. The recording must be 0.5-30 seconds, no larger than+50 MiB, and readable by libsndfile. The service converts it to mono and 44.1+kHz automatically.++Start the local service and open <http://127.0.0.1:8024>. If there are no voice+profiles, the page opens the registration view first.++```bash+bash start_server.sh+```++Registration is also available over HTTP:++```bash+curl http://127.0.0.1:8024/api/voices/register \+ -F 'audio=@/absolute/path/reference.wav' \+ -F 'text=The exact transcript of the reference recording.' \+ -F 'name=speaker_a' \+ -F 'overwrite=false'+```++The encoder is loaded only during registration. The online inference runtime+is released first and restored after registration completes.++## Inference++Command line:++```bash+bash run_infer.sh \+ --text "Welcome to Audio8 TTS ONNX Runtime." \+ --voice speaker_a \+ --max-new-tokens 256 \+ --output outputs/example.wav+```++The command also writes `outputs/example.npy` containing generated codec codes+with shape `[10, T]`.++HTTP WAV response:++```bash+curl http://127.0.0.1:8024/api/tts \+ -H 'Content-Type: application/json' \+ -d '{"text":"Welcome to Audio8 TTS.","voice_name":"speaker_a","max_new_tokens":256}' \+ -o outputs/api.wav+```++OpenAI-compatible WAV response:++```bash+curl http://127.0.0.1:8024/v1/audio/speech \+ -H 'Content-Type: application/json' \+ -d '{"model":"arktts","input":"Welcome to Audio8 TTS.","voice":"speaker_a","response_format":"wav"}' \+ -o outputs/openai.wav+```++`POST /api/tts/stream` returns newline-delimited JSON. Audio chunks contain+base64-encoded signed 16-bit little-endian mono PCM at 44.1 kHz. Cancel the+active stream with `POST /api/tts/cancel`.++The service serializes synthesis and registration requests to bound memory+use. It is intended for local use and low-concurrency CPU deployments.++## Configuration++| Variable | Default | Purpose |+|---|---|---|+| `ARKTTS_MODEL_DIR` | `onnx_runtime/model` | Downloaded Hugging Face model |+| `ARKTTS_VOICES_DIR` | `onnx_runtime/voices` | Registered voice profiles |+| `ARKTTS_REGISTRATION_DIR` | `$ARKTTS_MODEL_DIR/registration` | Codec encoder directory |+| `ARKTTS_THREADS` | `5` | ONNX Runtime CPU threads |+| `HOST` | `127.0.0.1` | Service listen address |+| `PORT` | `8024` | Service listen port |++Stop a managed background service with `bash stop_server.sh`. Logs are written+to `service.log`.++## Responsible use++The reference transcript must match the spoken content. Noisy, long, or+incorrectly transcribed references can reduce stability and speaker+similarity. Obtain consent before cloning a voice and disclose synthetic audio+where appropriate. Evaluate accuracy, safety, and legal compliance before+deployment.++The code and model weights are released under the Apache License 2.0. See the+repository [LICENSE](../LICENSE) and [NOTICE](../NOTICE).diff --git a/onnx_runtime/README_zh.md b/onnx_runtime/README_zh.mdnew file mode 100644index 0000000..58c05e2--- /dev/null+++ b/onnx_runtime/README_zh.md@@ -0,0 +1,140 @@+# Audio8 TTS ONNX Runtime++这是+[Audio8-TTS-Preview-0.6B-ONNX-INT4](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4)+的纯 CPU ONNX Runtime 推理实现,包含命令行推理、网页与 HTTP 服务、流式 PCM+输出和参考音频音色注册。模型下载完成后,运行时不依赖 PyTorch、Transformers+或 Hugging Face Hub。++英文文档:[README.md](README.md)++## 精度与资源占用++| 组件 | 精度 |+|---|---|+| Slow/Fast AR 权重 | Weight-only INT4 |+| Activation、hidden state 和 KV cache | FP16 |+| Codec encoder/decoder | FP16 |+| 波形输出 | FP32 |++普通合成只加载 Slow AR、Fast AR 和 codec decoder 三个 session。在 16 GB+Apple M2 MacBook Air、5 个 ONNX Runtime 线程的测试环境中,服务加载后约占+1004 MiB,合成峰值约 1.1-1.2 GiB。注册音色前会释放在线模型,再单独加载+encoder;实测注册峰值约 1.55 GiB。不同系统和 ONNX Runtime 内存分配器下的+数值可能有所变化。++在线推理模型文件约 572 MiB;加上可选的音色注册 encoder,完整下载约 968 MiB。++## 安装++需要 Python 3.11 或更高版本。当前版本已在 macOS arm64 的+`CPUExecutionProvider` 上验证。++```bash+git clone https://github.com/Audio8-AI/Audio8_TTS.git+cd Audio8_TTS/onnx_runtime++python3 -m pip install -U "huggingface_hub[cli]"+hf download Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4 --local-dir model+bash setup.sh+```++模型目录结构应为:++```text+model/+|- slow_ar_int4.onnx(.data)+|- fast_ar_int4.onnx(.data)+|- codec_decoder_fp16.onnx(.data)+|- runtime_manifest.json+|- tokenizer/tokenizer.json+`- registration/+ |- codec_encoder_fp16.onnx(.data)+ `- registration_manifest.json+```++模型放在其他位置时设置 `ARKTTS_MODEL_DIR`。音色默认写入 `voices/`,可用+`ARKTTS_VOICES_DIR` 修改。++## 注册音色++音色 profile 由参考音频的 codec codes 和准确原文组成。参考音频必须为+0.5-30 秒、不超过 50 MiB,并且可由 libsndfile 读取。服务会自动转为单声道并+重采样到 44.1 kHz。++启动服务并打开 <http://127.0.0.1:8024>。没有音色时,页面会优先显示注册界面。++```bash+bash start_server.sh+```++也可以通过 HTTP 注册:++```bash+curl http://127.0.0.1:8024/api/voices/register \+ -F 'audio=@/absolute/path/reference.wav' \+ -F 'text=参考音频对应的准确原文' \+ -F 'name=speaker_a' \+ -F 'overwrite=false'+```++Encoder 只在注册期间加载。注册前会释放在线推理模型,完成后自动恢复。++## 推理++命令行:++```bash+bash run_infer.sh \+ --text "你好,这是 Audio8 TTS ONNX Runtime。" \+ --voice speaker_a \+ --max-new-tokens 256 \+ --output outputs/example.wav+```++命令同时写入 `[10,T]` 形状的 `outputs/example.npy` codec codes。++HTTP WAV:++```bash+curl http://127.0.0.1:8024/api/tts \+ -H 'Content-Type: application/json' \+ -d '{"text":"你好,这是 Audio8 TTS。","voice_name":"speaker_a","max_new_tokens":256}' \+ -o outputs/api.wav+```++OpenAI 兼容接口:++```bash+curl http://127.0.0.1:8024/v1/audio/speech \+ -H 'Content-Type: application/json' \+ -d '{"model":"arktts","input":"你好,这是 Audio8 TTS。","voice":"speaker_a","response_format":"wav"}' \+ -o outputs/openai.wav+```++`POST /api/tts/stream` 返回 NDJSON,其中音频块为 44.1 kHz、单声道、signed+16-bit little-endian PCM 的 Base64 数据。使用 `POST /api/tts/cancel` 取消当前流。++服务会串行处理合成和注册请求以限制内存占用,定位是本地使用和低并发 CPU 部署。++## 配置++| 环境变量 | 默认值 | 用途 |+|---|---|---|+| `ARKTTS_MODEL_DIR` | `onnx_runtime/model` | Hugging Face 模型目录 |+| `ARKTTS_VOICES_DIR` | `onnx_runtime/voices` | 已注册音色目录 |+| `ARKTTS_REGISTRATION_DIR` | `$ARKTTS_MODEL_DIR/registration` | Codec encoder 目录 |+| `ARKTTS_THREADS` | `5` | ONNX Runtime CPU 线程数 |+| `HOST` | `127.0.0.1` | 服务监听地址 |+| `PORT` | `8024` | 服务端口 |++使用 `bash stop_server.sh` 停止后台服务,日志写入 `service.log`。++## 负责任使用++参考原文必须与录音内容一致。噪声、过长或原文错误的参考音频会降低稳定性和+说话人相似度。克隆声音前应取得授权,并在适当场景明确披露合成音频。部署前请+评估准确性、安全性和法律合规性。++代码和模型权重采用 Apache License 2.0,详见仓库的 [LICENSE](../LICENSE) 和+[NOTICE](../NOTICE)。diff --git a/onnx_runtime/arktts_runtime/__init__.py b/onnx_runtime/arktts_runtime/__init__.pynew file mode 100644index 0000000..afef7ac--- /dev/null+++ b/onnx_runtime/arktts_runtime/__init__.py@@ -0,0 +1,5 @@+"""ArkTTS ONNX runtime package.++Import ``ArkTtsRuntime`` from ``arktts_runtime.runtime`` so tokenizer-only tools do+not eagerly require ONNX Runtime.+"""diff --git a/onnx_runtime/arktts_runtime/cli.py b/onnx_runtime/arktts_runtime/cli.pynew file mode 100644index 0000000..e38e212--- /dev/null+++ b/onnx_runtime/arktts_runtime/cli.py@@ -0,0 +1,54 @@+from __future__ import annotations++import argparse+from pathlib import Path++import numpy as np+import soundfile as sf++from .runtime import ArkTtsRuntime++ONNX_ROOT = Path(__file__).resolve().parents[1]+++def main() -> None:+ parser = argparse.ArgumentParser(description="Audio8 TTS ONNX Runtime")+ parser.add_argument("--model-dir", type=Path, default=ONNX_ROOT / "model")+ parser.add_argument("--voices-dir", type=Path, default=ONNX_ROOT / "voices")+ parser.add_argument("--precision", choices=["int4"])+ parser.add_argument("--codec-precision", choices=["fp16"])+ parser.add_argument("--text", required=True)+ parser.add_argument("--voice", required=True)+ parser.add_argument("--output", type=Path, default=Path("output.wav"))+ parser.add_argument("--max-new-tokens", type=int, default=1024)+ parser.add_argument("--temperature", type=float, default=0.7)+ parser.add_argument("--top-p", type=float, default=0.9)+ parser.add_argument("--top-k", type=int, default=50)+ parser.add_argument("--seed", type=int, default=42)+ parser.add_argument("--threads", type=int, default=5)+ args = parser.parse_args()+ runtime = ArkTtsRuntime(+ args.model_dir,+ args.voices_dir,+ args.precision,+ args.codec_precision,+ args.threads,+ )+ audio, codes = runtime.synthesize(+ text=args.text,+ voice=args.voice,+ max_new_tokens=args.max_new_tokens,+ temperature=args.temperature,+ top_p=args.top_p,+ top_k=args.top_k,+ seed=args.seed,+ )+ args.output.parent.mkdir(parents=True, exist_ok=True)+ sf.write(str(args.output), audio, int(runtime.manifest["sample_rate"]))+ np.save(args.output.with_suffix(".npy"), codes.astype(np.uint16))+ print(f"saved {args.output}")+ print(f"saved {args.output.with_suffix('.npy')}")+++if __name__ == "__main__":+ main()diff --git a/onnx_runtime/arktts_runtime/prompt.py b/onnx_runtime/arktts_runtime/prompt.pynew file mode 100644index 0000000..da4eed0--- /dev/null+++ b/onnx_runtime/arktts_runtime/prompt.py@@ -0,0 +1,57 @@+from __future__ import annotations++import re+from pathlib import Path++import numpy as np+from tokenizers import Tokenizer+++def clean_text(text: str) -> str:+ return " ".join(str(text).strip().split())+++def format_reference_text(text: str) -> str:+ text = clean_text(text)+ return text if re.search(r"<\|speaker:\d+\|>", text) else f"<|speaker:0|>{text}"+++class PromptBuilder:+ def __init__(self, tokenizer_dir: Path, semantic_begin_id: int, num_codebooks: int):+ self.tokenizer = Tokenizer.from_file(str(tokenizer_dir / "tokenizer.json"))+ self.semantic_begin_id = int(semantic_begin_id)+ self.num_codebooks = int(num_codebooks)++ def encode_text(self, text: str) -> list[int]:+ return list(self.tokenizer.encode(text, add_special_tokens=False).ids)++ def build(+ self, target_text: str, reference_text: str, reference_codes: np.ndarray+ ) -> np.ndarray:+ codes = np.asarray(reference_codes, dtype=np.int64)+ if codes.ndim != 2 or codes.shape[0] != self.num_codebooks or codes.shape[1] == 0:+ raise ValueError(+ f"reference codes must have shape [{self.num_codebooks}, T>0], got {codes.shape}"+ )+ prefix_parts = [+ "<|im_start|>system\n",+ "convert the provided text to speech reference to the following:\n\nText:\n",+ format_reference_text(reference_text),+ "\n\nSpeech:\n",+ ]+ suffix_parts = [+ "<|im_end|>\n",+ "<|im_start|>user\n",+ clean_text(target_text),+ "<|im_end|>\n",+ "<|im_start|>assistant\n<|voice|>",+ ]+ prefix = [token for part in prefix_parts for token in self.encode_text(part)]+ suffix = [token for part in suffix_parts for token in self.encode_text(part)]+ semantic_ids = (codes[0] + self.semantic_begin_id).tolist()+ row0 = np.asarray(prefix + semantic_ids + suffix, dtype=np.int64)+ values = np.zeros((self.num_codebooks + 1, row0.size), dtype=np.int64)+ values[0] = row0+ begin = len(prefix)+ values[1:, begin : begin + codes.shape[1]] = codes+ return values[np.newaxis]diff --git a/onnx_runtime/arktts_runtime/registration.py b/onnx_runtime/arktts_runtime/registration.pynew file mode 100644index 0000000..7b1f70a--- /dev/null+++ b/onnx_runtime/arktts_runtime/registration.py@@ -0,0 +1,142 @@+from __future__ import annotations++import gc+import hashlib+import io+import json+import os+import shutil+import tempfile+from datetime import datetime, timezone+from math import gcd+from pathlib import Path++import numpy as np+import onnxruntime as ort+import soundfile as sf++MAX_AUDIO_BYTES = 50 * 1024 * 1024+MAX_AUDIO_SECONDS = 30.0+MIN_AUDIO_SECONDS = 0.5+++class VoiceRegistration:+ def __init__(self, model_dir: Path, voices_root: Path, model_fingerprint: str):+ self.root = model_dir.resolve()+ self.voices_root = voices_root.resolve()+ self.encoder_path = self.root / "codec_encoder_fp16.onnx"+ self.manifest_path = self.root / "registration_manifest.json"+ self.expected_fingerprint = model_fingerprint++ def status(self) -> dict:+ available = self.encoder_path.is_file() and self.manifest_path.is_file()+ reason = None+ if available:+ try:+ manifest = json.loads(self.manifest_path.read_text())+ if manifest.get("model_fingerprint") != self.expected_fingerprint:+ available = False+ reason = "registration model fingerprint does not match online model"+ except (OSError, json.JSONDecodeError):+ available = False+ reason = "registration manifest is invalid"+ else:+ reason = "registration package was not found"+ return {"available": available, "reason": reason}++ @staticmethod+ def _validate_name(name: str) -> str:+ value = name.strip()+ if not value or len(value) > 64 or value in {".", ".."} or Path(value).name != value:+ raise ValueError("voice name must be one path component with at most 64 characters")+ return value++ @staticmethod+ def _decode_audio(data: bytes, target_rate: int) -> tuple[np.ndarray, int]:+ if not data or len(data) > MAX_AUDIO_BYTES:+ raise ValueError("audio file must be between 1 byte and 50 MiB")+ try:+ audio, sample_rate = sf.read(io.BytesIO(data), dtype="float32", always_2d=True)+ except (RuntimeError, sf.LibsndfileError) as exc:+ raise ValueError(f"unsupported or invalid audio file: {exc}") from exc+ audio = audio.mean(axis=1)+ duration = audio.size / max(1, int(sample_rate))+ if duration < MIN_AUDIO_SECONDS or duration > MAX_AUDIO_SECONDS:+ raise ValueError("reference audio duration must be between 0.5 and 30 seconds")+ if not np.isfinite(audio).all():+ raise ValueError("reference audio contains non-finite samples")+ if int(sample_rate) != int(target_rate):+ from scipy.signal import resample_poly++ factor = gcd(int(sample_rate), int(target_rate))+ audio = resample_poly(audio, target_rate // factor, int(sample_rate) // factor).astype(+ np.float32+ )+ padding = (-audio.size) % 2048+ if padding:+ audio = np.pad(audio, (0, padding))+ return np.ascontiguousarray(audio.reshape(1, 1, -1)), int(sample_rate)++ def register(self, data: bytes, filename: str, text: str, name: str, overwrite: bool) -> dict:+ state = self.status()+ if not state["available"]:+ raise RuntimeError(state["reason"])+ voice_name = self._validate_name(name)+ reference_text = " ".join(text.strip().split())+ if not reference_text:+ raise ValueError("reference text must not be empty")+ manifest = json.loads(self.manifest_path.read_text())+ audio, source_rate = self._decode_audio(data, int(manifest["sample_rate"]))+ target = self.voices_root / voice_name+ if target.exists() and not overwrite:+ raise FileExistsError(f"voice already exists: {voice_name}")++ options = ort.SessionOptions()+ options.log_severity_level = 3+ session = None+ try:+ session = ort.InferenceSession(+ str(self.encoder_path),+ sess_options=options,+ providers=["CPUExecutionProvider"],+ )+ input_type = session.get_inputs()[0].type+ values = audio.astype(np.float16 if input_type == "tensor(float16)" else np.float32)+ codes = np.asarray(session.run(None, {"audio": values})[0], dtype=np.int64)+ finally:+ del session+ gc.collect()+ if codes.ndim == 3:+ codes = codes[0]+ if (+ codes.ndim != 2+ or codes.shape[0] != int(manifest["num_codebooks"])+ or codes.shape[1] == 0+ ):+ raise RuntimeError(f"encoder returned invalid codes: {codes.shape}")++ self.voices_root.mkdir(parents=True, exist_ok=True)+ temp = Path(tempfile.mkdtemp(prefix=f".{voice_name}.", dir=self.voices_root))+ try:+ np.save(temp / "codes.npy", codes.astype(np.uint16))+ meta = {+ "name": voice_name,+ "reference_text": reference_text,+ "shape": list(codes.shape),+ "dtype": "uint16",+ "sample_rate": int(manifest["sample_rate"]),+ "source_audio": Path(filename or "reference_audio").name,+ "source_sample_rate": source_rate,+ "source_sha256": hashlib.sha256(data).hexdigest(),+ "model_fingerprint": manifest["model_fingerprint"],+ "created_at": datetime.now(timezone.utc).isoformat(),+ "source_kind": "web_registration",+ }+ (temp / "meta.json").write_text(json.dumps(meta, ensure_ascii=False, indent=2) + "\n")+ if target.exists():+ shutil.rmtree(target)+ os.replace(temp, target)+ finally:+ if temp.exists():+ shutil.rmtree(temp)+ return metadiff --git a/onnx_runtime/arktts_runtime/runtime.py b/onnx_runtime/arktts_runtime/runtime.pynew file mode 100644index 0000000..6cfa6ca--- /dev/null+++ b/onnx_runtime/arktts_runtime/runtime.py@@ -0,0 +1,281 @@+from __future__ import annotations++import json+import threading+from collections.abc import Iterator+from pathlib import Path++import numpy as np+import onnxruntime as ort++from .prompt import PromptBuilder+from .voices import VoiceStore++ORT_DTYPES = {+ "tensor(float)": np.float32,+ "tensor(float16)": np.float16,+ "tensor(int64)": np.int64,+ "tensor(bool)": np.bool_,+}+++def _session(path: Path, threads: int | None = None) -> ort.InferenceSession:+ options = ort.SessionOptions()+ options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL+ options.log_severity_level = 3+ if threads is not None:+ options.intra_op_num_threads = int(threads)+ options.inter_op_num_threads = max(1, int(threads) // 2)+ return ort.InferenceSession(str(path), sess_options=options, providers=["CPUExecutionProvider"])+++def _sample(logits: np.ndarray, temperature: float, top_p: float, top_k: int, rng) -> int:+ values = np.asarray(logits, dtype=np.float64).reshape(-1)+ order = np.argsort(values)[::-1]+ sorted_values = values[order]+ base = np.exp(sorted_values - np.max(sorted_values))+ base /= base.sum()+ cumulative = np.cumsum(base)+ remove = (cumulative > float(top_p)) | (np.arange(base.size) >= int(top_k))+ remove[0] = False+ masked = values.copy()+ masked[order[remove]] = -np.inf+ scaled = masked / max(float(temperature), 1e-5)+ scaled -= np.max(scaled)+ probs = np.exp(scaled)+ probs /= probs.sum()+ noise = -np.log(np.clip(rng.random(probs.size), 1e-12, 1.0))+ return int(np.argmax(probs / noise))+++class ArkTtsRuntime:+ def __init__(+ self,+ model_dir: Path,+ voices_dir: Path,+ precision: str | None = None,+ codec_precision: str | None = None,+ threads: int | None = None,+ ):+ self.model_dir = Path(model_dir).resolve()+ self.manifest = json.loads((self.model_dir / "runtime_manifest.json").read_text())+ self.precision = precision or self.manifest["default_precision"]+ if self.precision not in self.manifest["available_precisions"]:+ raise ValueError(f"unsupported precision: {self.precision}")+ self.codec_precision = codec_precision or self.manifest.get(+ "default_codec_precision", "fp16"+ )+ available_codec = self.manifest.get("available_codec_precisions", ["fp16"])+ if self.codec_precision not in available_codec:+ raise ValueError(f"unsupported codec precision: {self.codec_precision}")+ self.slow = _session(+ self.model_dir / f"slow_ar_{self.precision}.onnx",+ threads,+ )+ self.fast = _session(+ self.model_dir / f"fast_ar_{self.precision}.onnx",+ threads,+ )+ codec_models = self.manifest.get("codec_models", {"fp16": "codec_decoder_fp16.onnx"})+ self.decoder = _session(+ self.model_dir / codec_models[self.codec_precision],+ threads,+ )+ self.prompt_builder = PromptBuilder(+ self.model_dir / "tokenizer",+ self.manifest["semantic_begin_id"],+ self.manifest["num_codebooks"],+ )+ self.voices = VoiceStore(Path(voices_dir), self.manifest["num_codebooks"])+ self.slow_inputs = {item.name: item for item in self.slow.get_inputs()}+ self.fast_inputs = {item.name: item for item in self.fast.get_inputs()}++ def _empty_slow_caches(self) -> list[np.ndarray]:+ dtype = ORT_DTYPES[self.slow_inputs["cache_key_0"].type]+ shape = (+ 1,+ int(self.manifest["n_local_heads"]),+ int(self.manifest["max_seq_len"]),+ int(self.manifest["head_dim"]),+ )+ return [np.zeros(shape, dtype=dtype) for _ in range(2 * int(self.manifest["num_layers"]))]++ def _empty_fast_caches(self) -> list[np.ndarray]:+ dtype = ORT_DTYPES[self.fast_inputs["cache_key_0"].type]+ shape = (+ 1,+ int(self.manifest["fast_n_local_heads"]),+ int(self.manifest["num_codebooks"]),+ int(self.manifest["fast_head_dim"]),+ )+ return [+ np.zeros(shape, dtype=dtype) for _ in range(2 * int(self.manifest["num_fast_layers"]))+ ]++ @staticmethod+ def _update_caches(+ caches: list[np.ndarray], positions: np.ndarray, deltas: list[np.ndarray]+ ) -> None:+ for index, delta in enumerate(deltas):+ caches[index][:, :, positions, :] = delta++ def _slow_step(self, codes: np.ndarray, positions: np.ndarray, caches):+ feeds = {"codes": codes.astype(np.int64), "input_pos": positions.astype(np.int64)}+ for i in range(int(self.manifest["num_layers"])):+ feeds[f"cache_key_{i}"] = caches[2 * i]+ feeds[f"cache_value_{i}"] = caches[2 * i + 1]+ outputs = self.slow.run(None, feeds)+ self._update_caches(caches, positions, outputs[2:])+ return np.asarray(outputs[0])[0, -1], np.asarray(outputs[1])[:, -1:, :]++ def _fast_step(self, hidden, token_id: int, use_hidden: bool, position: int, caches):+ hidden_dtype = ORT_DTYPES[self.fast_inputs["slow_hidden"].type]+ feeds = {+ "slow_hidden": np.asarray(hidden, dtype=hidden_dtype),+ "token_id": np.asarray([[token_id]], dtype=np.int64),+ "use_slow_hidden": np.asarray([use_hidden], dtype=np.bool_),+ "input_pos": np.asarray([position], dtype=np.int64),+ }+ for i in range(int(self.manifest["num_fast_layers"])):+ feeds[f"cache_key_{i}"] = caches[2 * i]+ feeds[f"cache_value_{i}"] = caches[2 * i + 1]+ outputs = self.fast.run(None, feeds)+ self._update_caches(caches, np.asarray([position]), outputs[1:])+ return np.asarray(outputs[0])[0, -1]++ def _sample_semantic(self, logits, previous, temperature, top_p, top_k, rng):+ begin = int(self.manifest["semantic_begin_id"])+ end = int(self.manifest["semantic_end_id"])+ stop = int(self.manifest["im_end_id"])+ allowed_ids = np.concatenate([np.arange(begin, end + 1), np.asarray([stop])])+ values = np.asarray(logits).reshape(-1)+ allowed_logits = (+ values+ if self.manifest.get("slow_logits_layout") == "semantic_then_eos"+ else values[allowed_ids]+ )+ if allowed_logits.size != allowed_ids.size:+ raise ValueError(+ f"unexpected slow logits size: {allowed_logits.size}, expected {allowed_ids.size}"+ )+ normal_index = _sample(allowed_logits, temperature, top_p, top_k, rng)+ normal = int(allowed_ids[normal_index])+ high_index = _sample(allowed_logits, 1.0, 0.9, top_k, rng)+ high = int(allowed_ids[high_index])+ if begin <= normal <= end and normal in previous:+ return high+ return normal++ def iter_codes(+ self,+ text: str,+ voice: str,+ max_new_tokens: int = 1024,+ temperature: float = 0.7,+ top_p: float = 0.9,+ top_k: int = 50,+ seed: int = 42,+ stop_event: threading.Event | None = None,+ ) -> Iterator[np.ndarray]:+ reference_codes, meta = self.voices.load(voice)+ prompt = self.prompt_builder.build(text, meta["reference_text"], reference_codes)+ prompt_len = int(prompt.shape[2])+ max_seq_len = int(self.manifest["max_seq_len"])+ if prompt_len >= max_seq_len:+ raise ValueError(+ f"prompt length {prompt_len} exceeds max sequence length {max_seq_len}"+ )+ max_new_tokens = min(int(max_new_tokens), max_seq_len - prompt_len)+ rng = np.random.default_rng(int(seed))+ slow_caches = self._empty_slow_caches()+ positions = np.arange(prompt_len, dtype=np.int64)+ logits, hidden = self._slow_step(prompt, positions, slow_caches)+ previous: list[int] = []+ begin = int(self.manifest["semantic_begin_id"])+ stop = int(self.manifest["im_end_id"])+ codebook_size = int(self.manifest["codebook_size"])++ for step in range(max_new_tokens):+ if stop_event is not None and stop_event.is_set():+ return+ semantic = self._sample_semantic(logits, previous, temperature, top_p, top_k, rng)+ if semantic == stop:+ return+ previous.append(semantic)+ previous = previous[-10:]+ fast_caches = self._empty_fast_caches()+ self._fast_step(hidden, 0, True, 0, fast_caches)+ token = min(max(semantic - begin, 0), codebook_size - 1)+ codebooks = [token]+ for fast_pos in range(1, int(self.manifest["num_codebooks"])):+ fast_logits = self._fast_step(hidden, token, False, fast_pos, fast_caches)+ token = _sample(fast_logits, temperature, top_p, top_k, rng)+ codebooks.append(token)+ frame = np.asarray(codebooks, dtype=np.int64)+ yield frame+ if step + 1 >= max_new_tokens:+ return+ column = np.concatenate([[semantic], frame]).reshape(1, -1, 1)+ position = np.asarray([prompt_len + step], dtype=np.int64)+ logits, hidden = self._slow_step(column, position, slow_caches)++ def decode_codes(self, codes: np.ndarray) -> np.ndarray:+ values = np.asarray(codes, dtype=np.int64)+ if values.ndim == 2:+ values = values[np.newaxis]+ if values.ndim != 3 or values.shape[1] != int(self.manifest["num_codebooks"]):+ raise ValueError(f"invalid generated codes shape: {values.shape}")+ audio = self.decoder.run(None, {"codes": values})[0]+ return np.asarray(audio, dtype=np.float32).reshape(-1)++ def synthesize(self, **kwargs) -> tuple[np.ndarray, np.ndarray]:+ frames = list(self.iter_codes(**kwargs))+ if not frames:+ raise RuntimeError("model produced no codec frames")+ codes = np.stack(frames, axis=1)+ return self.decode_codes(codes), codes++ def stream(self, chunk_frames: int = 12, **kwargs):+ all_frames: list[np.ndarray] = []+ emitted_samples = 0+ hop = int(self.manifest["codec_hop_length"])+ context = int(self.manifest.get("stream_context_frames", 128))+ guard = int(self.manifest.get("stream_guard_frames", 1)) * hop+ seq = 0+ for frame in self.iter_codes(**kwargs):+ all_frames.append(frame)+ if len(all_frames) % int(chunk_frames) != 0:+ continue+ start_frame = max(0, len(all_frames) - context - int(chunk_frames))+ window = np.stack(all_frames[start_frame:], axis=1)+ audio = self.decode_codes(window)+ absolute_start = start_frame * hop+ stable_end = absolute_start + max(0, audio.size - guard)+ begin = max(0, emitted_samples - absolute_start)+ end = max(begin, stable_end - absolute_start)+ if end > begin:+ chunk = np.ascontiguousarray(audio[begin:end])+ emitted_samples += chunk.size+ yield {+ "type": "audio_chunk",+ "seq": seq,+ "audio": chunk,+ "frame_count": len(all_frames),+ }+ seq += 1+ if not all_frames:+ raise RuntimeError("model produced no codec frames")+ start_frame = max(0, len(all_frames) - context - int(chunk_frames))+ window = np.stack(all_frames[start_frame:], axis=1)+ audio = self.decode_codes(window)+ absolute_start = start_frame * hop+ begin = max(0, emitted_samples - absolute_start)+ if begin < audio.size:+ chunk = np.ascontiguousarray(audio[begin:])+ yield {+ "type": "audio_chunk",+ "seq": seq,+ "audio": chunk,+ "frame_count": len(all_frames),+ }+ yield {"type": "complete", "codes": np.stack(all_frames, axis=1)}diff --git a/onnx_runtime/arktts_runtime/service.py b/onnx_runtime/arktts_runtime/service.pynew file mode 100644index 0000000..5d4d253--- /dev/null+++ b/onnx_runtime/arktts_runtime/service.py@@ -0,0 +1,614 @@+from __future__ import annotations++# ruff: noqa: B008, E501+import base64+import ctypes+import gc+import json+import os+import re+import subprocess+import threading+import time+from pathlib import Path++import numpy as np+import soundfile as sf+from fastapi import FastAPI, File, Form, HTTPException, UploadFile+from fastapi.responses import HTMLResponse, Response, StreamingResponse+from pydantic import BaseModel, Field++from .registration import MAX_AUDIO_BYTES, VoiceRegistration+from .runtime import ArkTtsRuntime++ROOT = Path(__file__).resolve().parents[1]+MODEL_DIR = Path(os.environ.get("ARKTTS_MODEL_DIR", ROOT / "model")).resolve()+VOICES_DIR = Path(os.environ.get("ARKTTS_VOICES_DIR", ROOT / "voices")).resolve()+PRECISION = os.environ.get("ARKTTS_PRECISION")+CODEC_PRECISION = os.environ.get("ARKTTS_CODEC_PRECISION")+THREADS = int(os.environ.get("ARKTTS_THREADS", "5"))+REGISTRATION_DIR = Path(+ os.environ.get(+ "ARKTTS_REGISTRATION_DIR",+ MODEL_DIR / "registration",+ )+).resolve()+runtime: ArkTtsRuntime | None = None+registration: VoiceRegistration | None = None+request_lock = threading.Lock()+registration_lock = threading.Lock()+reload_lock = threading.Lock()+active_stop: threading.Event | None = None+service_started_at = time.monotonic()+memory_lock = threading.Lock()+memory_cache: dict[str, float | None] = {"sampled_at": 0.0, "current_mb": None, "peak_mb": None}+app = FastAPI(title="Audio8 TTS ONNX Runtime")+++TEST_PAGE = """<!doctype html>+<html lang="zh-CN">+<head>+ <meta charset="utf-8">+ <meta name="viewport" content="width=device-width, initial-scale=1">+ <title>Audio8 TTS 本地测试</title>+ <style>+ :root { color-scheme: light; --ink:#18201f; --muted:#65706e; --line:#d7dddb; --paper:#f5f7f6; --accent:#087f5b; --accent-dark:#066447; --danger:#b42318; }+ * { box-sizing:border-box; }+ body { margin:0; min-width:320px; background:var(--paper); color:var(--ink); font:15px/1.5 -apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif; letter-spacing:0; }+ header { border-bottom:1px solid var(--line); background:#fff; }+ .bar, main { width:min(880px, calc(100% - 32px)); margin:auto; }+ .bar { min-height:64px; display:flex; align-items:center; justify-content:space-between; gap:16px; }+ h1 { margin:0; font-size:20px; font-weight:680; }+ .header-actions, .health { display:flex; align-items:center; gap:8px; }+ .health { color:var(--muted); font-size:13px; }+ .icon-button { width:36px; height:36px; padding:0; border:1px solid var(--line); border-radius:6px; background:#fff; color:var(--ink); font:22px/34px -apple-system,BlinkMacSystemFont,sans-serif; }+ .icon-button:hover { border-color:#aeb8b5; background:#f7f9f8; }+ .icon-button:disabled { opacity:.45; }+ .dot { width:9px; height:9px; border-radius:50%; background:#a3aaa8; }+ .dot.ok { background:#12a371; }+ main { padding:32px 0 48px; }+ .meta { display:flex; flex-wrap:wrap; gap:8px 18px; margin-bottom:18px; color:var(--muted); font-size:13px; }+ .metrics { display:grid; grid-template-columns:repeat(3, 1fr); border-top:1px solid var(--line); border-bottom:1px solid var(--line); margin-bottom:24px; }+ .metric { min-width:0; padding:15px 0; }+ .metric + .metric { padding-left:18px; border-left:1px solid var(--line); }+ .metric span { display:block; color:var(--muted); font-size:12px; }+ .metric strong { display:block; margin-top:2px; font-size:19px; font-weight:680; font-variant-numeric:tabular-nums; }+ .memory-track { grid-column:1 / -1; height:3px; margin-top:-3px; overflow:hidden; background:#dfe5e3; }+ .memory-fill { width:0; height:100%; background:var(--accent); transition:width .35s ease; }+ .tabs { display:flex; width:100%; margin-bottom:24px; border-bottom:1px solid var(--line); }+ .tab { height:42px; padding:0 18px; border:0; border-bottom:2px solid transparent; border-radius:0; background:transparent; color:var(--muted); }+ .tab:hover { background:transparent; color:var(--ink); }+ .tab.active { border-bottom-color:var(--accent); color:var(--ink); }+ .view[hidden] { display:none; }+ label { display:block; margin:0 0 7px; font-size:13px; font-weight:650; }+ textarea, select, input[type=number], input[type=text], input[type=file] { width:100%; border:1px solid #bfc8c5; border-radius:6px; background:#fff; color:var(--ink); font:inherit; outline:none; }+ textarea:focus, select:focus, input:focus { border-color:var(--accent); box-shadow:0 0 0 3px rgba(8,127,91,.12); }+ textarea { min-height:156px; resize:vertical; padding:13px 14px; }+ select, input[type=number], input[type=text], input[type=file] { height:42px; padding:0 11px; }+ input[type=file] { padding:8px; }+ .row { display:grid; grid-template-columns:1.35fr repeat(3, 1fr); gap:14px; margin-top:18px; }+ .actions { display:flex; align-items:center; gap:10px; margin-top:24px; }+ button, .download { height:42px; border-radius:6px; padding:0 17px; font:650 14px/40px inherit; cursor:pointer; text-decoration:none; }+ button { border:1px solid var(--accent); background:var(--accent); color:#fff; }+ button:hover { background:var(--accent-dark); }+ button:disabled { cursor:not-allowed; opacity:.55; }+ .download { display:none; border:1px solid #bfc8c5; background:#fff; color:var(--ink); }+ .status { min-height:22px; margin:18px 0 10px; color:var(--muted); font-size:13px; }+ .status.error { color:var(--danger); }+ audio { display:none; width:100%; margin-top:10px; }+ audio.ready { display:block; }+ .register-grid { display:grid; grid-template-columns:1fr 1fr; gap:14px; margin-bottom:18px; }+ .register-text { min-height:112px; }+ .check { display:flex; align-items:center; gap:8px; margin:14px 0 0; font-weight:500; }+ .check input { width:16px; height:16px; accent-color:var(--accent); }+ .audio-preview { display:none; margin:14px 0 0; }+ .audio-preview.ready { display:block; }+ @media (max-width:700px) { .row { grid-template-columns:1fr 1fr; } main { padding-top:24px; } }+ @media (max-width:440px) { .row, .register-grid { grid-template-columns:1fr; } .metrics { grid-template-columns:1fr 1fr; } .metric:nth-child(3) { grid-column:1 / -1; padding-left:0; border-left:0; border-top:1px solid var(--line); } .actions { align-items:stretch; flex-direction:column; } button, .download { width:100%; text-align:center; } }+ </style>+</head>+<body>+ <header><div class="bar"><h1>Audio8 TTS</h1><div class="header-actions"><div class="health"><span id="dot" class="dot"></span><span id="health">正在加载模型</span></div><button id="reloadRuntime" class="icon-button" type="button" title="重新加载模型" aria-label="重新加载模型">↻</button></div></div></header>+ <main>+ <div class="meta"><span id="precision">DualAR:-</span><span id="codecPrecision">Codec:-</span><span id="provider">执行器:-</span></div>+ <section class="metrics" aria-label="运行状态">+ <div class="metric"><span>当前内存</span><strong id="memoryCurrent">读取中</strong></div>+ <div class="metric"><span>峰值内存</span><strong id="memoryPeak">读取中</strong></div>+ <div class="metric"><span>运行时间</span><strong id="uptime">-</strong></div>+ <div class="memory-track"><div id="memoryFill" class="memory-fill"></div></div>+ </section>+ <div class="tabs" role="tablist"><button id="ttsTab" class="tab active" role="tab">语音合成</button><button id="registerTab" class="tab" role="tab">音频注册</button></div>+ <section id="ttsView" class="view">+ <label for="text">合成文本</label>+ <textarea id="text">你好,这是 Audio8 TTS 的本地语音合成测试。</textarea>+ <div class="row">+ <div><label for="voice">音色</label><select id="voice"></select></div>+ <div><label for="temperature">Temperature</label><input id="temperature" type="number" value="0.3" min="0.05" max="2" step="0.05"></div>+ <div><label for="topP">Top P</label><input id="topP" type="number" value="0.9" min="0.05" max="1" step="0.05"></div>+ <div><label for="seed">Seed</label><input id="seed" type="number" value="42" min="0" step="1"></div>+ </div>+ <div class="actions"><button id="generate" disabled>生成语音</button><a id="download" class="download" download="audio8_output.wav">下载 WAV</a></div>+ <div id="status" class="status"></div>+ <audio id="audio" controls></audio>+ </section>+ <section id="registerView" class="view" hidden>+ <div class="register-grid">+ <div><label for="referenceAudio">参考音频</label><input id="referenceAudio" type="file" accept="audio/*,.wav,.flac,.mp3,.m4a"></div>+ <div><label for="voiceName">音色名称</label><input id="voiceName" type="text" maxlength="64" placeholder="speaker_a"></div>+ </div>+ <label for="referenceText">参考音频原文</label>+ <textarea id="referenceText" class="register-text"></textarea>+ <audio id="referencePreview" class="audio-preview" controls></audio>+ <label class="check"><input id="overwriteVoice" type="checkbox">覆盖同名音色</label>+ <div class="actions"><button id="registerVoice" disabled>注册音色</button></div>+ <div id="registerStatus" class="status"></div>+ </section>+ </main>+ <script>+ const $ = id => document.getElementById(id);+ let audioUrl = null;+ let referenceUrl = null;+ let registrationAvailable = false;+ function selectView(name) {+ const register = name === 'register';+ $('ttsView').hidden = register;+ $('registerView').hidden = !register;+ $('ttsTab').classList.toggle('active', !register);+ $('registerTab').classList.toggle('active', register);+ }+ async function refreshVoices(selected) {+ const response = await fetch('/api/voices');+ if (!response.ok) throw new Error('读取音色失败');+ const data = await response.json();+ const names = data.voices.map(voice => typeof voice === 'string' ? voice : voice.name);+ $('voice').replaceChildren(...names.map(name => new Option(name, name)));+ if (selected && names.includes(selected)) $('voice').value = selected;+ $('generate').disabled = names.length === 0;+ return names;+ }+ function formatUptime(seconds) {+ const value = Math.max(0, Math.floor(seconds));+ const hours = Math.floor(value / 3600);+ const minutes = Math.floor((value % 3600) / 60);+ const secs = value % 60;+ return hours ? `${hours}时 ${minutes}分 ${secs}秒` : `${minutes}分 ${secs}秒`;+ }+ async function refreshMetrics() {+ try {+ const response = await fetch('/api/system', {cache:'no-store'});+ if (!response.ok) throw new Error(`HTTP ${response.status}`);+ const data = await response.json();+ $('memoryCurrent').textContent = data.memory.current_mb == null ? '不可用' : `${(data.memory.current_mb / 1024).toFixed(2)} GB`;+ $('memoryPeak').textContent = data.memory.peak_mb == null ? '不可用' : `${(data.memory.peak_mb / 1024).toFixed(2)} GB`;+ $('uptime').textContent = formatUptime(data.uptime_seconds);+ const scale = Math.max(4096, data.memory.peak_mb || 0);+ $('memoryFill').style.width = `${Math.min(100, ((data.memory.current_mb || 0) / scale) * 100)}%`;+ } catch (_) {+ $('memoryCurrent').textContent = '读取失败';+ }+ }+ async function initialize() {+ try {+ const [health, registrationStatus] = await Promise.all([fetch('/api/health'), fetch('/api/registration/status')]);+ if (!health.ok || !registrationStatus.ok) throw new Error('服务初始化失败');+ const healthData = await health.json();+ const registrationData = await registrationStatus.json();+ $('precision').textContent = `DualAR:${healthData.precision.toUpperCase()}`;+ $('codecPrecision').textContent = `Codec:${healthData.codec_precision.toUpperCase()}`;+ $('provider').textContent = `执行器:${healthData.providers.slow[0]}`;+ const voiceNames = await refreshVoices();+ registrationAvailable = registrationData.available;+ $('registerVoice').disabled = !registrationAvailable;+ $('registerStatus').textContent = registrationAvailable ? '' : registrationData.reason;+ $('dot').classList.add('ok');+ $('health').textContent = '模型已就绪';+ $('generate').disabled = voiceNames.length === 0;+ $('status').textContent = voiceNames.length ? '' : '没有可用音色';+ if (voiceNames.length === 0 && registrationAvailable) selectView('register');+ } catch (error) {+ $('health').textContent = '服务不可用';+ $('status').textContent = error.message;+ $('status').classList.add('error');+ }+ }+ $('generate').addEventListener('click', async () => {+ const text = $('text').value.trim();+ if (!text) { $('status').textContent = '请输入合成文本'; return; }+ $('generate').disabled = true;+ $('status').classList.remove('error');+ $('status').textContent = '正在生成语音...';+ $('audio').classList.remove('ready');+ $('download').style.display = 'none';+ const started = performance.now();+ try {+ const response = await fetch('/api/tts', {+ method:'POST', headers:{'Content-Type':'application/json'},+ body:JSON.stringify({text, voice_name:$('voice').value, temperature:Number($('temperature').value), top_p:Number($('topP').value), seed:Number($('seed').value)})+ });+ if (!response.ok) throw new Error((await response.text()) || `HTTP ${response.status}`);+ const blob = await response.blob();+ if (audioUrl) URL.revokeObjectURL(audioUrl);+ audioUrl = URL.createObjectURL(blob);+ $('audio').src = audioUrl;+ $('audio').classList.add('ready');+ $('download').href = audioUrl;+ $('download').style.display = 'inline-block';+ $('status').textContent = `生成完成,用时 ${((performance.now()-started)/1000).toFixed(2)} 秒`;+ await $('audio').play().catch(() => {});+ } catch (error) {+ $('status').textContent = `生成失败:${error.message}`;+ $('status').classList.add('error');+ } finally { $('generate').disabled = false; }+ });+ $('ttsTab').addEventListener('click', () => selectView('tts'));+ $('registerTab').addEventListener('click', () => selectView('register'));+ $('referenceAudio').addEventListener('change', () => {+ const file = $('referenceAudio').files[0];+ if (referenceUrl) URL.revokeObjectURL(referenceUrl);+ if (!file) { $('referencePreview').classList.remove('ready'); return; }+ referenceUrl = URL.createObjectURL(file);+ $('referencePreview').src = referenceUrl;+ $('referencePreview').classList.add('ready');+ });+ $('registerVoice').addEventListener('click', async () => {+ const file = $('referenceAudio').files[0];+ const name = $('voiceName').value.trim();+ const text = $('referenceText').value.trim();+ if (!file || !name || !text) { $('registerStatus').textContent = '请选择音频并填写音色名称和准确原文'; return; }+ $('registerVoice').disabled = true;+ $('registerStatus').classList.remove('error');+ $('registerStatus').textContent = '正在注册音色...';+ const form = new FormData();+ form.append('audio', file);+ form.append('name', name);+ form.append('text', text);+ form.append('overwrite', $('overwriteVoice').checked ? 'true' : 'false');+ try {+ const response = await fetch('/api/voices/register', {method:'POST', body:form});+ if (!response.ok) throw new Error((await response.json()).detail || `HTTP ${response.status}`);+ const result = await response.json();+ await refreshVoices(result.voice.name);+ $('registerStatus').textContent = `注册完成:${result.voice.name}(${result.voice.shape[1]} 帧)`;+ } catch (error) {+ $('registerStatus').textContent = `注册失败:${error.message}`;+ $('registerStatus').classList.add('error');+ } finally { $('registerVoice').disabled = !registrationAvailable; }+ });+ $('reloadRuntime').addEventListener('click', async () => {+ const button = $('reloadRuntime');+ button.disabled = true;+ $('dot').classList.remove('ok');+ $('health').textContent = '正在重新加载模型';+ try {+ const response = await fetch('/api/runtime/reload', {method:'POST'});+ const result = await response.json();+ if (!response.ok) throw new Error(result.detail || `HTTP ${response.status}`);+ $('precision').textContent = `DualAR:${result.precision.toUpperCase()}`;+ $('codecPrecision').textContent = `Codec:${result.codec_precision.toUpperCase()}`;+ await refreshVoices();+ await refreshMetrics();+ $('dot').classList.add('ok');+ $('health').textContent = `模型已重新加载(${result.reload_seconds.toFixed(2)} 秒)`;+ } catch (error) {+ $('health').textContent = `重新加载失败:${error.message}`;+ } finally { button.disabled = false; }+ });+ initialize();+ refreshMetrics();+ setInterval(refreshMetrics, 2000);+ </script>+</body>+</html>"""+++class TtsRequest(BaseModel):+ text: str = Field(..., min_length=1, max_length=1000)+ voice_name: str+ max_new_tokens: int = Field(1024, ge=16, le=2048)+ temperature: float = Field(0.3, gt=0.0, le=2.0)+ top_p: float = Field(0.9, gt=0.0, le=1.0)+ top_k: int = Field(50, ge=1, le=4096)+ seed: int = 42+++class OpenAiSpeechRequest(BaseModel):+ model: str = "arktts"+ input: str = Field(..., min_length=1, max_length=1000)+ voice: str+ response_format: str = "wav"++[email protected]("/", response_class=HTMLResponse)+def test_page():+ return TEST_PAGE++[email protected]_event("startup")+def startup() -> None:+ global registration, runtime+ runtime = ArkTtsRuntime(MODEL_DIR, VOICES_DIR, PRECISION, CODEC_PRECISION, THREADS)+ registration = VoiceRegistration(+ REGISTRATION_DIR,+ runtime.voices.root,+ str(runtime.manifest["model_fingerprint"]),+ )+++def require_runtime() -> ArkTtsRuntime:+ if runtime is None:+ raise HTTPException(503, "runtime is not loaded")+ return runtime+++def require_registration() -> VoiceRegistration:+ if registration is None:+ raise HTTPException(503, "registration is not initialized")+ return registration+++def release_allocator_memory() -> None:+ gc.collect()+ if os.uname().sysname != "Darwin":+ return+ try:+ system = ctypes.CDLL("/usr/lib/libSystem.B.dylib")+ system.malloc_zone_pressure_relief.argtypes = [ctypes.c_void_p, ctypes.c_size_t]+ system.malloc_zone_pressure_relief.restype = ctypes.c_size_t+ system.malloc_zone_pressure_relief(None, 0)+ except (AttributeError, OSError):+ pass+++def clear_runtime_cache() -> None:+ with memory_lock:+ memory_cache.update(sampled_at=0.0, current_mb=None, peak_mb=None)+++def _memory_value_mb(output: str, pattern: str) -> float | None:+ match = re.search(pattern, output, flags=re.IGNORECASE)+ if match is None:+ return None+ value = float(match.group(1))+ unit = match.group(2).upper()+ return value * {"KB": 1 / 1024, "MB": 1, "GB": 1024}[unit]+++def memory_stats() -> dict[str, float | None]:+ now = time.monotonic()+ with memory_lock:+ if now - float(memory_cache["sampled_at"] or 0.0) < 1.0:+ return memory_cache.copy()+ try:+ result = subprocess.run(+ ["/usr/bin/footprint", str(os.getpid())],+ capture_output=True,+ check=True,+ text=True,+ timeout=5,+ )+ current_mb = _memory_value_mb(result.stdout, r"Footprint:\s*([\d.]+)\s*(KB|MB|GB)")+ peak_mb = _memory_value_mb(+ result.stdout, r"phys_footprint_peak:\s*([\d.]+)\s*(KB|MB|GB)"+ )+ except (OSError, subprocess.SubprocessError):+ current_mb = None+ peak_mb = None+ memory_cache.update(sampled_at=now, current_mb=current_mb, peak_mb=peak_mb)+ return memory_cache.copy()++[email protected]("/api/health")+def health():+ obj = require_runtime()+ return {+ "ok": True,+ "precision": obj.precision,+ "codec_precision": obj.codec_precision,+ "providers": {+ "slow": obj.slow.get_providers(),+ "fast": obj.fast.get_providers(),+ "decoder": obj.decoder.get_providers(),+ },+ }++[email protected]("/api/voices")+def voices():+ return {"voices": require_runtime().voices.list()}++[email protected]("/api/registration/status")+def registration_status():+ return require_registration().status()++[email protected]("/api/voices/register")+def register_voice(+ audio: UploadFile = File(...),+ text: str = Form(...),+ name: str = Form(...),+ overwrite: bool = Form(False),+):+ global runtime+ data = audio.file.read(MAX_AUDIO_BYTES + 1)+ try:+ with registration_lock, request_lock:+ current = require_runtime()+ precision = current.precision+ codec_precision = current.codec_precision+ runtime = None+ del current+ clear_runtime_cache()+ release_allocator_memory()+ try:+ meta = require_registration().register(+ data,+ audio.filename or "reference_audio",+ text,+ name,+ overwrite,+ )+ finally:+ release_allocator_memory()+ runtime = ArkTtsRuntime(MODEL_DIR, VOICES_DIR, precision, codec_precision, THREADS)+ clear_runtime_cache()+ except FileExistsError as exc:+ raise HTTPException(409, str(exc)) from exc+ except ValueError as exc:+ raise HTTPException(400, str(exc)) from exc+ except RuntimeError as exc:+ raise HTTPException(503, str(exc)) from exc+ return {"ok": True, "voice": meta}++[email protected]("/api/system")+def system_stats():+ stats = memory_stats()+ return {+ "memory": {"current_mb": stats["current_mb"], "peak_mb": stats["peak_mb"]},+ "uptime_seconds": time.monotonic() - service_started_at,+ "refresh_interval_ms": 2000,+ }++[email protected]("/api/runtime/reload")+def reload_runtime():+ global active_stop, registration, runtime+ if active_stop is not None:+ active_stop.set()+ started = time.monotonic()+ with reload_lock, request_lock:+ old_runtime = runtime+ precision = old_runtime.precision if old_runtime is not None else PRECISION+ codec_precision = (+ old_runtime.codec_precision if old_runtime is not None else CODEC_PRECISION+ )+ runtime = None+ registration = None+ del old_runtime+ clear_runtime_cache()+ release_allocator_memory()+ try:+ runtime = ArkTtsRuntime(MODEL_DIR, VOICES_DIR, precision, codec_precision, THREADS)+ registration = VoiceRegistration(+ REGISTRATION_DIR,+ runtime.voices.root,+ str(runtime.manifest["model_fingerprint"]),+ )+ except Exception as exc:+ raise HTTPException(500, f"failed to reload runtime: {exc}") from exc+ return {+ "ok": True,+ "precision": runtime.precision,+ "codec_precision": runtime.codec_precision,+ "reload_seconds": time.monotonic() - started,+ }++[email protected]("/api/tts/stream")+def stream_tts(request: TtsRequest):+ def events():+ global active_stop+ stop = threading.Event()+ active_stop = stop+ try:+ with request_lock:+ obj = require_runtime()+ yield (+ json.dumps(+ {+ "event": "start",+ "sample_rate": int(obj.manifest["sample_rate"]),+ "channels": 1,+ "sample_format": "s16le",+ "precision": obj.precision,+ "codec_precision": obj.codec_precision,+ }+ )+ + "\n"+ )+ for event in obj.stream(+ text=request.text,+ voice=request.voice_name,+ max_new_tokens=request.max_new_tokens,+ temperature=request.temperature,+ top_p=request.top_p,+ top_k=request.top_k,+ seed=request.seed,+ stop_event=stop,+ ):+ if stop.is_set():+ yield json.dumps({"event": "cancelled"}) + "\n"+ return+ if event["type"] == "audio_chunk":+ pcm = (np.clip(event["audio"], -1.0, 1.0) * 32767.0).astype("<i2")+ yield (+ json.dumps(+ {+ "event": "audio_chunk",+ "seq": int(event["seq"]),+ "frame_count": int(event["frame_count"]),+ "pcm_b64": base64.b64encode(pcm.tobytes()).decode("ascii"),+ }+ )+ + "\n"+ )+ else:+ yield (+ json.dumps(+ {"event": "complete", "frame_count": int(event["codes"].shape[1])}+ )+ + "\n"+ )+ finally:+ if active_stop is stop:+ active_stop = None++ return StreamingResponse(events(), media_type="application/x-ndjson")++[email protected]("/api/tts/cancel")+def cancel():+ if active_stop is None:+ return {"ok": True, "cancelled": False}+ active_stop.set()+ return {"ok": True, "cancelled": True}++[email protected]("/v1/audio/speech")+def openai_speech(request: OpenAiSpeechRequest):+ if request.model not in {"arktts", "tts-1"}:+ raise HTTPException(400, f"unsupported model: {request.model}")+ if request.response_format not in {"wav", "pcm"}:+ raise HTTPException(400, "response_format must be wav or pcm")+ with request_lock:+ obj = require_runtime()+ audio, _ = obj.synthesize(text=request.input, voice=request.voice)+ if request.response_format == "pcm":+ pcm = (np.clip(audio, -1.0, 1.0) * 32767.0).astype("<i2").tobytes()+ return Response(pcm, media_type="application/octet-stream")+ import io++ buffer = io.BytesIO()+ sf.write(buffer, audio, int(obj.manifest["sample_rate"]), format="WAV")+ return Response(buffer.getvalue(), media_type="audio/wav")++[email protected]("/api/tts")+def tts(request: TtsRequest):+ with request_lock:+ obj = require_runtime()+ audio, _ = obj.synthesize(+ text=request.text,+ voice=request.voice_name,+ max_new_tokens=request.max_new_tokens,+ temperature=request.temperature,+ top_p=request.top_p,+ top_k=request.top_k,+ seed=request.seed,+ )+ import io++ buffer = io.BytesIO()+ sf.write(buffer, audio, int(obj.manifest["sample_rate"]), format="WAV")+ return Response(buffer.getvalue(), media_type="audio/wav")diff --git a/onnx_runtime/arktts_runtime/voices.py b/onnx_runtime/arktts_runtime/voices.pynew file mode 100644index 0000000..74985c2--- /dev/null+++ b/onnx_runtime/arktts_runtime/voices.py@@ -0,0 +1,40 @@+from __future__ import annotations++import json+from pathlib import Path++import numpy as np+++class VoiceStore:+ def __init__(self, root: Path, num_codebooks: int):+ self.root = root.resolve()+ self.num_codebooks = int(num_codebooks)++ def list(self) -> list[dict]:+ voices = []+ if not self.root.exists():+ return voices+ for meta_path in sorted(self.root.glob("*/meta.json")):+ try:+ voices.append(json.loads(meta_path.read_text()))+ except (OSError, json.JSONDecodeError):+ continue+ return voices++ def load(self, name: str) -> tuple[np.ndarray, dict]:+ if not name or Path(name).name != name:+ raise ValueError("invalid voice name")+ voice_dir = self.root / name+ meta_path = voice_dir / "meta.json"+ codes_path = voice_dir / "codes.npy"+ if not meta_path.is_file() or not codes_path.is_file():+ raise KeyError(f"voice not found: {name}")+ meta = json.loads(meta_path.read_text())+ codes = np.load(codes_path, allow_pickle=False).astype(np.int64, copy=False)+ if codes.ndim != 2 or codes.shape[0] != self.num_codebooks or codes.shape[1] == 0:+ raise ValueError(f"invalid codes for voice {name}: {codes.shape}")+ reference_text = str(meta.get("reference_text", "")).strip()+ if not reference_text:+ raise ValueError(f"voice {name} has no reference_text")+ return codes, metadiff --git a/onnx_runtime/requirements.txt b/onnx_runtime/requirements.txtnew file mode 100644index 0000000..308e373--- /dev/null+++ b/onnx_runtime/requirements.txt@@ -0,0 +1,8 @@+numpy==2.4.3+onnxruntime==1.24.4+soundfile==0.13.1+scipy==1.17.1+fastapi==0.135.2+uvicorn==0.42.0+pydantic>=2.9,<3+python-multipart==0.0.22diff --git a/onnx_runtime/run_infer.sh b/onnx_runtime/run_infer.shnew file mode 100755index 0000000..f652811--- /dev/null+++ b/onnx_runtime/run_infer.sh@@ -0,0 +1,8 @@+#!/usr/bin/env bash+set -euo pipefail++ROOT="$(cd "$(dirname "$0")" && pwd)"+exec "$ROOT/.venv/bin/python" -m arktts_runtime.cli \+ --model-dir "${ARKTTS_MODEL_DIR:-$ROOT/model}" \+ --voices-dir "${ARKTTS_VOICES_DIR:-$ROOT/voices}" \+ "$@"diff --git a/onnx_runtime/run_server.sh b/onnx_runtime/run_server.shnew file mode 100755index 0000000..b8f7eeb--- /dev/null+++ b/onnx_runtime/run_server.sh@@ -0,0 +1,15 @@+#!/usr/bin/env bash+set -euo pipefail++ROOT="$(cd "$(dirname "$0")" && pwd)"+export ARKTTS_MODEL_DIR="${ARKTTS_MODEL_DIR:-$ROOT/model}"+export ARKTTS_VOICES_DIR="${ARKTTS_VOICES_DIR:-$ROOT/voices}"+export ARKTTS_REGISTRATION_DIR="${ARKTTS_REGISTRATION_DIR:-$ARKTTS_MODEL_DIR/registration}"+export ARKTTS_PRECISION="int4"+export ARKTTS_CODEC_PRECISION="fp16"+export ARKTTS_THREADS="${ARKTTS_THREADS:-5}"++exec "$ROOT/.venv/bin/uvicorn" arktts_runtime.service:app \+ --app-dir "$ROOT" \+ --host "${HOST:-127.0.0.1}" \+ --port "${PORT:-8024}"diff --git a/onnx_runtime/setup.sh b/onnx_runtime/setup.shnew file mode 100755index 0000000..4f17fda--- /dev/null+++ b/onnx_runtime/setup.sh@@ -0,0 +1,23 @@+#!/usr/bin/env bash+set -euo pipefail++ROOT="$(cd "$(dirname "$0")" && pwd)"+PYTHON_BIN="${PYTHON_BIN:-python3}"++if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then+ echo "Python was not found. Set PYTHON_BIN to a Python 3.11+ executable." >&2+ exit 1+fi++if ! "$PYTHON_BIN" -c 'import sys; raise SystemExit(sys.version_info < (3, 11))'; then+ echo "Python 3.11 or newer is required." >&2+ exit 1+fi++"$PYTHON_BIN" -m venv "$ROOT/.venv"+"$ROOT/.venv/bin/python" -m pip install --upgrade pip+"$ROOT/.venv/bin/python" -m pip install -r "$ROOT/requirements.txt"+# Only the local Rust tokenizer runtime is needed. Its optional Hub dependency+# tree (huggingface-hub, httpx, hf-xet, and friends) is not used here.+"$ROOT/.venv/bin/python" -m pip install --no-deps tokenizers==0.22.2+echo "Environment ready: $ROOT/.venv"diff --git a/onnx_runtime/start_server.sh b/onnx_runtime/start_server.shnew file mode 100755index 0000000..7d58c60--- /dev/null+++ b/onnx_runtime/start_server.sh@@ -0,0 +1,30 @@+#!/usr/bin/env bash+set -euo pipefail++ROOT="$(cd "$(dirname "$0")" && pwd)"+PID_FILE="$ROOT/.service.pid"+PORT="${PORT:-8024}"++if [[ -f "$PID_FILE" ]] && kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then+ echo "Service is already running with PID $(cat "$PID_FILE")"+ exit 0+fi++PORT="$PORT" nohup "$ROOT/run_server.sh" >"$ROOT/service.log" 2>&1 &+PID=$!+echo "$PID" >"$PID_FILE"++for _ in {1..60}; do+ if curl -fsS "http://127.0.0.1:$PORT/api/health" >/dev/null 2>&1; then+ echo "Audio8 TTS is ready: http://127.0.0.1:$PORT"+ exit 0+ fi+ if ! kill -0 "$PID" 2>/dev/null; then+ echo "Service exited during startup. See $ROOT/service.log" >&2+ exit 1+ fi+ sleep 1+done++echo "Service did not become ready in 60 seconds. See $ROOT/service.log" >&2+exit 1diff --git a/onnx_runtime/stop_server.sh b/onnx_runtime/stop_server.shnew file mode 100755index 0000000..3963bff--- /dev/null+++ b/onnx_runtime/stop_server.sh@@ -0,0 +1,23 @@+#!/usr/bin/env bash+set -euo pipefail++ROOT="$(cd "$(dirname "$0")" && pwd)"+PID_FILE="$ROOT/.service.pid"++if [[ ! -f "$PID_FILE" ]]; then+ echo "No managed Audio8 TTS service is running."+ exit 0+fi++PID="$(cat "$PID_FILE")"+if kill -0 "$PID" 2>/dev/null; then+ kill "$PID"+ for _ in {1..30}; do+ if ! kill -0 "$PID" 2>/dev/null; then+ break+ fi+ sleep 1+ done+fi+rm -f "$PID_FILE"+echo "Audio8 TTS service stopped."