Skip to content

SOTA-Class TTS at Compact Scale

Python

Add CPU-only ONNX INT4 runtime

linyubupa5f90e9fb82ce
diff --git a/.gitignore b/.gitignore
index ce29964..6d4454a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -11,6 +11,11 @@ checkpoints/
prepared_data/
*.log
+/onnx_runtime/model/
+/onnx_runtime/voices/
+/onnx_runtime/outputs/
+/onnx_runtime/.service.pid
+
# Model weights are distributed separately from the source repository.
/model/
/configs/
diff --git a/README.md b/README.md
index 8e17f33..0fe1aae 100644
--- a/README.md
+++ b/README.md
@@ -8,7 +8,8 @@
[![GitHub](https://img.shields.io/badge/GitHub-Audio8__TTS-black?style=for-the-badge&logo=github)](https://github.com/Audio8-AI/Audio8_TTS)
[![Demo](https://img.shields.io/badge/Demo-Audio%20Samples-brightgreen?style=for-the-badge&logo=githubpages)](https://audio8-ai.github.io/Audio8_TTS/)
-[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Audio8--TTS--Preview--0.6b-yellow?style=for-the-badge)](https://huggingface.co/AutoArk-AI/Audio8-TTS-Preview-0.6b)
+[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Audio8--TTS--Preview--0.6b-yellow?style=for-the-badge)](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6b)
+[![ONNX INT4](https://img.shields.io/badge/ONNX-INT4-005CED?style=for-the-badge&logo=onnx)](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue?style=for-the-badge)](LICENSE)
中文文档: [README_zh.md](README_zh.md)
@@ -73,7 +74,7 @@ pip install -r requirements.txt
```
Download the checkpoint from
-[Hugging Face](https://huggingface.co/AutoArk-AI/Audio8-TTS-Preview-0.6b) and
+[Hugging Face](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6b) and
place it in the repository's `model/` directory. The expected local checkpoint
path is `model/audio8_tts_0_6B_preview/`. All commands also accept a Hugging
Face model ID through `--model`.
@@ -122,6 +123,21 @@ The batch command writes `manifest.jsonl` and `failures.jsonl`. Existing WAV
files are skipped unless `--overwrite` is passed. See
`python audio8_tts_infer.py --help` for sampling and code-saving options.
+## CPU ONNX Runtime
+
+[`onnx_runtime/`](onnx_runtime/) provides a standalone CPU deployment using
+weight-only INT4 Slow/Fast AR models, FP16 activations and KV caches, and an
+FP16 codec. It includes CLI inference, a local web and HTTP service, streaming
+PCM output, and reference-voice registration without PyTorch or Transformers.
+
+The online sessions use about 1 GiB of memory in the tested Apple M2 setup.
+During voice registration, the online sessions are released before the codec
+encoder is loaded to keep peak memory low.
+
+Download the ONNX model from
+[Audio8-TTS-Preview-0.6B-ONNX-INT4](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4)
+and follow the [ONNX Runtime guide](onnx_runtime/README.md).
+
## SGLang Omni Serving
The adapter in [`sglang_omni/`](sglang_omni/) provides an OpenAI-compatible
diff --git a/README_zh.md b/README_zh.md
index e356e3b..d43fdab 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -8,7 +8,8 @@
[![GitHub](https://img.shields.io/badge/GitHub-Audio8__TTS-black?style=for-the-badge&logo=github)](https://github.com/Audio8-AI/Audio8_TTS)
[![Demo](https://img.shields.io/badge/Demo-Audio%20Samples-brightgreen?style=for-the-badge&logo=githubpages)](https://audio8-ai.github.io/Audio8_TTS/)
-[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Audio8--TTS--Preview--0.6b-yellow?style=for-the-badge)](https://huggingface.co/AutoArk-AI/Audio8-TTS-Preview-0.6b)
+[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Audio8--TTS--Preview--0.6b-yellow?style=for-the-badge)](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6b)
+[![ONNX INT4](https://img.shields.io/badge/ONNX-INT4-005CED?style=for-the-badge&logo=onnx)](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue?style=for-the-badge)](LICENSE)
English: [README.md](README.md)
@@ -112,6 +113,20 @@ python audio8_tts_infer.py \
批量命令会生成 `manifest.jsonl` 和 `failures.jsonl`。除非传入 `--overwrite`,否则已有 WAV 会
被跳过。采样和 codec codes 保存参数可通过 `python audio8_tts_infer.py --help` 查看。
+## CPU ONNX Runtime
+
+[`onnx_runtime/`](onnx_runtime/) 提供独立的 CPU 部署方案:Slow/Fast AR 使用
+weight-only INT4,activation 和 KV cache 使用 FP16,codec 使用 FP16。它包含
+命令行推理、本地网页与 HTTP 服务、流式 PCM 输出和参考音频音色注册,运行时不依赖
+PyTorch 或 Transformers。
+
+在 Apple M2 测试环境中,在线模型 session 约占 1 GiB 内存。注册音色前会先释放
+在线 session,再单独加载 codec encoder,从而控制峰值内存。
+
+从
+[Audio8-TTS-Preview-0.6B-ONNX-INT4](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4)
+下载模型,并参阅 [ONNX Runtime 中文指南](onnx_runtime/README_zh.md)。
+
## SGLang Omni 服务部署
[`sglang_omni/`](sglang_omni/) 中的适配器提供兼容 OpenAI API 的推理服务,包含 SGLang
diff --git a/onnx_runtime/README.md b/onnx_runtime/README.md
new file mode 100644
index 0000000..f82dc30
--- /dev/null
+++ b/onnx_runtime/README.md
@@ -0,0 +1,151 @@
+# Audio8 TTS ONNX Runtime
+
+CPU-only ONNX Runtime inference for
+[Audio8-TTS-Preview-0.6B-ONNX-INT4](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4),
+including CLI inference, a web and HTTP service, streaming PCM output, and
+reference-voice registration. The runtime does not require PyTorch,
+Transformers, or Hugging Face Hub after the model files have been downloaded.
+
+Chinese documentation: [README_zh.md](README_zh.md)
+
+## Precision and resource use
+
+| Component | Precision |
+|---|---|
+| Slow/Fast AR weights | Weight-only INT4 |
+| Activations, hidden states, and KV cache | FP16 |
+| Codec encoder and decoder | FP16 |
+| Waveform output | FP32 |
+
+Normal synthesis loads only the Slow AR, Fast AR, and codec decoder sessions.
+On a 16 GB Apple M2 MacBook Air with five ONNX Runtime threads, the service
+used about 1004 MiB after loading and approximately 1.1-1.2 GiB at synthesis
+peak. Voice registration unloads those sessions before loading the encoder;
+the measured registration peak was approximately 1.55 GiB. Measurements vary
+by operating system and ONNX Runtime allocator behavior.
+
+The online model files occupy about 572 MiB. The optional voice-registration
+encoder brings the complete download to about 968 MiB.
+
+## Install
+
+Python 3.11 or newer is required. The current release is tested on macOS arm64
+with `CPUExecutionProvider`.
+
+```bash
+git clone https://github.com/Audio8-AI/Audio8_TTS.git
+cd Audio8_TTS/onnx_runtime
+
+python3 -m pip install -U "huggingface_hub[cli]"
+hf download Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4 --local-dir model
+bash setup.sh
+```
+
+The downloaded directory must have this layout:
+
+```text
+model/
+|- slow_ar_int4.onnx(.data)
+|- fast_ar_int4.onnx(.data)
+|- codec_decoder_fp16.onnx(.data)
+|- runtime_manifest.json
+|- tokenizer/tokenizer.json
+`- registration/
+ |- codec_encoder_fp16.onnx(.data)
+ `- registration_manifest.json
+```
+
+Set `ARKTTS_MODEL_DIR` if the model is stored elsewhere. Voice profiles are
+stored in `voices/` by default; override this with `ARKTTS_VOICES_DIR`.
+
+## Register a voice
+
+A voice profile contains codec codes extracted from a reference recording and
+its exact transcript. The recording must be 0.5-30 seconds, no larger than
+50 MiB, and readable by libsndfile. The service converts it to mono and 44.1
+kHz automatically.
+
+Start the local service and open <http://127.0.0.1:8024>. If there are no voice
+profiles, the page opens the registration view first.
+
+```bash
+bash start_server.sh
+```
+
+Registration is also available over HTTP:
+
+```bash
+curl http://127.0.0.1:8024/api/voices/register \
+ -F 'audio=@/absolute/path/reference.wav' \
+ -F 'text=The exact transcript of the reference recording.' \
+ -F 'name=speaker_a' \
+ -F 'overwrite=false'
+```
+
+The encoder is loaded only during registration. The online inference runtime
+is released first and restored after registration completes.
+
+## Inference
+
+Command line:
+
+```bash
+bash run_infer.sh \
+ --text "Welcome to Audio8 TTS ONNX Runtime." \
+ --voice speaker_a \
+ --max-new-tokens 256 \
+ --output outputs/example.wav
+```
+
+The command also writes `outputs/example.npy` containing generated codec codes
+with shape `[10, T]`.
+
+HTTP WAV response:
+
+```bash
+curl http://127.0.0.1:8024/api/tts \
+ -H 'Content-Type: application/json' \
+ -d '{"text":"Welcome to Audio8 TTS.","voice_name":"speaker_a","max_new_tokens":256}' \
+ -o outputs/api.wav
+```
+
+OpenAI-compatible WAV response:
+
+```bash
+curl http://127.0.0.1:8024/v1/audio/speech \
+ -H 'Content-Type: application/json' \
+ -d '{"model":"arktts","input":"Welcome to Audio8 TTS.","voice":"speaker_a","response_format":"wav"}' \
+ -o outputs/openai.wav
+```
+
+`POST /api/tts/stream` returns newline-delimited JSON. Audio chunks contain
+base64-encoded signed 16-bit little-endian mono PCM at 44.1 kHz. Cancel the
+active stream with `POST /api/tts/cancel`.
+
+The service serializes synthesis and registration requests to bound memory
+use. It is intended for local use and low-concurrency CPU deployments.
+
+## Configuration
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `ARKTTS_MODEL_DIR` | `onnx_runtime/model` | Downloaded Hugging Face model |
+| `ARKTTS_VOICES_DIR` | `onnx_runtime/voices` | Registered voice profiles |
+| `ARKTTS_REGISTRATION_DIR` | `$ARKTTS_MODEL_DIR/registration` | Codec encoder directory |
+| `ARKTTS_THREADS` | `5` | ONNX Runtime CPU threads |
+| `HOST` | `127.0.0.1` | Service listen address |
+| `PORT` | `8024` | Service listen port |
+
+Stop a managed background service with `bash stop_server.sh`. Logs are written
+to `service.log`.
+
+## Responsible use
+
+The reference transcript must match the spoken content. Noisy, long, or
+incorrectly transcribed references can reduce stability and speaker
+similarity. Obtain consent before cloning a voice and disclose synthetic audio
+where appropriate. Evaluate accuracy, safety, and legal compliance before
+deployment.
+
+The code and model weights are released under the Apache License 2.0. See the
+repository [LICENSE](../LICENSE) and [NOTICE](../NOTICE).
diff --git a/onnx_runtime/README_zh.md b/onnx_runtime/README_zh.md
new file mode 100644
index 0000000..58c05e2
--- /dev/null
+++ b/onnx_runtime/README_zh.md
@@ -0,0 +1,140 @@
+# Audio8 TTS ONNX Runtime
+
+这是
+[Audio8-TTS-Preview-0.6B-ONNX-INT4](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4)
+的纯 CPU ONNX Runtime 推理实现,包含命令行推理、网页与 HTTP 服务、流式 PCM
+输出和参考音频音色注册。模型下载完成后,运行时不依赖 PyTorch、Transformers
+或 Hugging Face Hub。
+
+英文文档:[README.md](README.md)
+
+## 精度与资源占用
+
+| 组件 | 精度 |
+|---|---|
+| Slow/Fast AR 权重 | Weight-only INT4 |
+| Activation、hidden state 和 KV cache | FP16 |
+| Codec encoder/decoder | FP16 |
+| 波形输出 | FP32 |
+
+普通合成只加载 Slow AR、Fast AR 和 codec decoder 三个 session。在 16 GB
+Apple M2 MacBook Air、5 个 ONNX Runtime 线程的测试环境中,服务加载后约占
+1004 MiB,合成峰值约 1.1-1.2 GiB。注册音色前会释放在线模型,再单独加载
+encoder;实测注册峰值约 1.55 GiB。不同系统和 ONNX Runtime 内存分配器下的
+数值可能有所变化。
+
+在线推理模型文件约 572 MiB;加上可选的音色注册 encoder,完整下载约 968 MiB。
+
+## 安装
+
+需要 Python 3.11 或更高版本。当前版本已在 macOS arm64 的
+`CPUExecutionProvider` 上验证。
+
+```bash
+git clone https://github.com/Audio8-AI/Audio8_TTS.git
+cd Audio8_TTS/onnx_runtime
+
+python3 -m pip install -U "huggingface_hub[cli]"
+hf download Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4 --local-dir model
+bash setup.sh
+```
+
+模型目录结构应为:
+
+```text
+model/
+|- slow_ar_int4.onnx(.data)
+|- fast_ar_int4.onnx(.data)
+|- codec_decoder_fp16.onnx(.data)
+|- runtime_manifest.json
+|- tokenizer/tokenizer.json
+`- registration/
+ |- codec_encoder_fp16.onnx(.data)
+ `- registration_manifest.json
+```
+
+模型放在其他位置时设置 `ARKTTS_MODEL_DIR`。音色默认写入 `voices/`,可用
+`ARKTTS_VOICES_DIR` 修改。
+
+## 注册音色
+
+音色 profile 由参考音频的 codec codes 和准确原文组成。参考音频必须为
+0.5-30 秒、不超过 50 MiB,并且可由 libsndfile 读取。服务会自动转为单声道并
+重采样到 44.1 kHz。
+
+启动服务并打开 <http://127.0.0.1:8024>。没有音色时,页面会优先显示注册界面。
+
+```bash
+bash start_server.sh
+```
+
+也可以通过 HTTP 注册:
+
+```bash
+curl http://127.0.0.1:8024/api/voices/register \
+ -F 'audio=@/absolute/path/reference.wav' \
+ -F 'text=参考音频对应的准确原文' \
+ -F 'name=speaker_a' \
+ -F 'overwrite=false'
+```
+
+Encoder 只在注册期间加载。注册前会释放在线推理模型,完成后自动恢复。
+
+## 推理
+
+命令行:
+
+```bash
+bash run_infer.sh \
+ --text "你好,这是 Audio8 TTS ONNX Runtime。" \
+ --voice speaker_a \
+ --max-new-tokens 256 \
+ --output outputs/example.wav
+```
+
+命令同时写入 `[10,T]` 形状的 `outputs/example.npy` codec codes。
+
+HTTP WAV:
+
+```bash
+curl http://127.0.0.1:8024/api/tts \
+ -H 'Content-Type: application/json' \
+ -d '{"text":"你好,这是 Audio8 TTS。","voice_name":"speaker_a","max_new_tokens":256}' \
+ -o outputs/api.wav
+```
+
+OpenAI 兼容接口:
+
+```bash
+curl http://127.0.0.1:8024/v1/audio/speech \
+ -H 'Content-Type: application/json' \
+ -d '{"model":"arktts","input":"你好,这是 Audio8 TTS。","voice":"speaker_a","response_format":"wav"}' \
+ -o outputs/openai.wav
+```
+
+`POST /api/tts/stream` 返回 NDJSON,其中音频块为 44.1 kHz、单声道、signed
+16-bit little-endian PCM 的 Base64 数据。使用 `POST /api/tts/cancel` 取消当前流。
+
+服务会串行处理合成和注册请求以限制内存占用,定位是本地使用和低并发 CPU 部署。
+
+## 配置
+
+| 环境变量 | 默认值 | 用途 |
+|---|---|---|
+| `ARKTTS_MODEL_DIR` | `onnx_runtime/model` | Hugging Face 模型目录 |
+| `ARKTTS_VOICES_DIR` | `onnx_runtime/voices` | 已注册音色目录 |
+| `ARKTTS_REGISTRATION_DIR` | `$ARKTTS_MODEL_DIR/registration` | Codec encoder 目录 |
+| `ARKTTS_THREADS` | `5` | ONNX Runtime CPU 线程数 |
+| `HOST` | `127.0.0.1` | 服务监听地址 |
+| `PORT` | `8024` | 服务端口 |
+
+使用 `bash stop_server.sh` 停止后台服务,日志写入 `service.log`。
+
+## 负责任使用
+
+参考原文必须与录音内容一致。噪声、过长或原文错误的参考音频会降低稳定性和
+说话人相似度。克隆声音前应取得授权,并在适当场景明确披露合成音频。部署前请
+评估准确性、安全性和法律合规性。
+
+代码和模型权重采用 Apache License 2.0,详见仓库的 [LICENSE](../LICENSE) 和
+[NOTICE](../NOTICE)。
diff --git a/onnx_runtime/arktts_runtime/__init__.py b/onnx_runtime/arktts_runtime/__init__.py
new file mode 100644
index 0000000..afef7ac
--- /dev/null
+++ b/onnx_runtime/arktts_runtime/__init__.py
@@ -0,0 +1,5 @@
+"""ArkTTS ONNX runtime package.
+
+Import ``ArkTtsRuntime`` from ``arktts_runtime.runtime`` so tokenizer-only tools do
+not eagerly require ONNX Runtime.
+"""
diff --git a/onnx_runtime/arktts_runtime/cli.py b/onnx_runtime/arktts_runtime/cli.py
new file mode 100644
index 0000000..e38e212
--- /dev/null
+++ b/onnx_runtime/arktts_runtime/cli.py
@@ -0,0 +1,54 @@
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+import numpy as np
+import soundfile as sf
+
+from .runtime import ArkTtsRuntime
+
+ONNX_ROOT = Path(__file__).resolve().parents[1]
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Audio8 TTS ONNX Runtime")
+ parser.add_argument("--model-dir", type=Path, default=ONNX_ROOT / "model")
+ parser.add_argument("--voices-dir", type=Path, default=ONNX_ROOT / "voices")
+ parser.add_argument("--precision", choices=["int4"])
+ parser.add_argument("--codec-precision", choices=["fp16"])
+ parser.add_argument("--text", required=True)
+ parser.add_argument("--voice", required=True)
+ parser.add_argument("--output", type=Path, default=Path("output.wav"))
+ parser.add_argument("--max-new-tokens", type=int, default=1024)
+ parser.add_argument("--temperature", type=float, default=0.7)
+ parser.add_argument("--top-p", type=float, default=0.9)
+ parser.add_argument("--top-k", type=int, default=50)
+ parser.add_argument("--seed", type=int, default=42)
+ parser.add_argument("--threads", type=int, default=5)
+ args = parser.parse_args()
+ runtime = ArkTtsRuntime(
+ args.model_dir,
+ args.voices_dir,
+ args.precision,
+ args.codec_precision,
+ args.threads,
+ )
+ audio, codes = runtime.synthesize(
+ text=args.text,
+ voice=args.voice,
+ max_new_tokens=args.max_new_tokens,
+ temperature=args.temperature,
+ top_p=args.top_p,
+ top_k=args.top_k,
+ seed=args.seed,
+ )
+ args.output.parent.mkdir(parents=True, exist_ok=True)
+ sf.write(str(args.output), audio, int(runtime.manifest["sample_rate"]))
+ np.save(args.output.with_suffix(".npy"), codes.astype(np.uint16))
+ print(f"saved {args.output}")
+ print(f"saved {args.output.with_suffix('.npy')}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/onnx_runtime/arktts_runtime/prompt.py b/onnx_runtime/arktts_runtime/prompt.py
new file mode 100644
index 0000000..da4eed0
--- /dev/null
+++ b/onnx_runtime/arktts_runtime/prompt.py
@@ -0,0 +1,57 @@
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+import numpy as np
+from tokenizers import Tokenizer
+
+
+def clean_text(text: str) -> str:
+ return " ".join(str(text).strip().split())
+
+
+def format_reference_text(text: str) -> str:
+ text = clean_text(text)
+ return text if re.search(r"<\|speaker:\d+\|>", text) else f"<|speaker:0|>{text}"
+
+
+class PromptBuilder:
+ def __init__(self, tokenizer_dir: Path, semantic_begin_id: int, num_codebooks: int):
+ self.tokenizer = Tokenizer.from_file(str(tokenizer_dir / "tokenizer.json"))
+ self.semantic_begin_id = int(semantic_begin_id)
+ self.num_codebooks = int(num_codebooks)
+
+ def encode_text(self, text: str) -> list[int]:
+ return list(self.tokenizer.encode(text, add_special_tokens=False).ids)
+
+ def build(
+ self, target_text: str, reference_text: str, reference_codes: np.ndarray
+ ) -> np.ndarray:
+ codes = np.asarray(reference_codes, dtype=np.int64)
+ if codes.ndim != 2 or codes.shape[0] != self.num_codebooks or codes.shape[1] == 0:
+ raise ValueError(
+ f"reference codes must have shape [{self.num_codebooks}, T>0], got {codes.shape}"
+ )
+ prefix_parts = [
+ "<|im_start|>system\n",
+ "convert the provided text to speech reference to the following:\n\nText:\n",
+ format_reference_text(reference_text),
+ "\n\nSpeech:\n",
+ ]
+ suffix_parts = [
+ "<|im_end|>\n",
+ "<|im_start|>user\n",
+ clean_text(target_text),
+ "<|im_end|>\n",
+ "<|im_start|>assistant\n<|voice|>",
+ ]
+ prefix = [token for part in prefix_parts for token in self.encode_text(part)]
+ suffix = [token for part in suffix_parts for token in self.encode_text(part)]
+ semantic_ids = (codes[0] + self.semantic_begin_id).tolist()
+ row0 = np.asarray(prefix + semantic_ids + suffix, dtype=np.int64)
+ values = np.zeros((self.num_codebooks + 1, row0.size), dtype=np.int64)
+ values[0] = row0
+ begin = len(prefix)
+ values[1:, begin : begin + codes.shape[1]] = codes
+ return values[np.newaxis]
diff --git a/onnx_runtime/arktts_runtime/registration.py b/onnx_runtime/arktts_runtime/registration.py
new file mode 100644
index 0000000..7b1f70a
--- /dev/null
+++ b/onnx_runtime/arktts_runtime/registration.py
@@ -0,0 +1,142 @@
+from __future__ import annotations
+
+import gc
+import hashlib
+import io
+import json
+import os
+import shutil
+import tempfile
+from datetime import datetime, timezone
+from math import gcd
+from pathlib import Path
+
+import numpy as np
+import onnxruntime as ort
+import soundfile as sf
+
+MAX_AUDIO_BYTES = 50 * 1024 * 1024
+MAX_AUDIO_SECONDS = 30.0
+MIN_AUDIO_SECONDS = 0.5
+
+
+class VoiceRegistration:
+ def __init__(self, model_dir: Path, voices_root: Path, model_fingerprint: str):
+ self.root = model_dir.resolve()
+ self.voices_root = voices_root.resolve()
+ self.encoder_path = self.root / "codec_encoder_fp16.onnx"
+ self.manifest_path = self.root / "registration_manifest.json"
+ self.expected_fingerprint = model_fingerprint
+
+ def status(self) -> dict:
+ available = self.encoder_path.is_file() and self.manifest_path.is_file()
+ reason = None
+ if available:
+ try:
+ manifest = json.loads(self.manifest_path.read_text())
+ if manifest.get("model_fingerprint") != self.expected_fingerprint:
+ available = False
+ reason = "registration model fingerprint does not match online model"
+ except (OSError, json.JSONDecodeError):
+ available = False
+ reason = "registration manifest is invalid"
+ else:
+ reason = "registration package was not found"
+ return {"available": available, "reason": reason}
+
+ @staticmethod
+ def _validate_name(name: str) -> str:
+ value = name.strip()
+ if not value or len(value) > 64 or value in {".", ".."} or Path(value).name != value:
+ raise ValueError("voice name must be one path component with at most 64 characters")
+ return value
+
+ @staticmethod
+ def _decode_audio(data: bytes, target_rate: int) -> tuple[np.ndarray, int]:
+ if not data or len(data) > MAX_AUDIO_BYTES:
+ raise ValueError("audio file must be between 1 byte and 50 MiB")
+ try:
+ audio, sample_rate = sf.read(io.BytesIO(data), dtype="float32", always_2d=True)
+ except (RuntimeError, sf.LibsndfileError) as exc:
+ raise ValueError(f"unsupported or invalid audio file: {exc}") from exc
+ audio = audio.mean(axis=1)
+ duration = audio.size / max(1, int(sample_rate))
+ if duration < MIN_AUDIO_SECONDS or duration > MAX_AUDIO_SECONDS:
+ raise ValueError("reference audio duration must be between 0.5 and 30 seconds")
+ if not np.isfinite(audio).all():
+ raise ValueError("reference audio contains non-finite samples")
+ if int(sample_rate) != int(target_rate):
+ from scipy.signal import resample_poly
+
+ factor = gcd(int(sample_rate), int(target_rate))
+ audio = resample_poly(audio, target_rate // factor, int(sample_rate) // factor).astype(
+ np.float32
+ )
+ padding = (-audio.size) % 2048
+ if padding:
+ audio = np.pad(audio, (0, padding))
+ return np.ascontiguousarray(audio.reshape(1, 1, -1)), int(sample_rate)
+
+ def register(self, data: bytes, filename: str, text: str, name: str, overwrite: bool) -> dict:
+ state = self.status()
+ if not state["available"]:
+ raise RuntimeError(state["reason"])
+ voice_name = self._validate_name(name)
+ reference_text = " ".join(text.strip().split())
+ if not reference_text:
+ raise ValueError("reference text must not be empty")
+ manifest = json.loads(self.manifest_path.read_text())
+ audio, source_rate = self._decode_audio(data, int(manifest["sample_rate"]))
+ target = self.voices_root / voice_name
+ if target.exists() and not overwrite:
+ raise FileExistsError(f"voice already exists: {voice_name}")
+
+ options = ort.SessionOptions()
+ options.log_severity_level = 3
+ session = None
+ try:
+ session = ort.InferenceSession(
+ str(self.encoder_path),
+ sess_options=options,
+ providers=["CPUExecutionProvider"],
+ )
+ input_type = session.get_inputs()[0].type
+ values = audio.astype(np.float16 if input_type == "tensor(float16)" else np.float32)
+ codes = np.asarray(session.run(None, {"audio": values})[0], dtype=np.int64)
+ finally:
+ del session
+ gc.collect()
+ if codes.ndim == 3:
+ codes = codes[0]
+ if (
+ codes.ndim != 2
+ or codes.shape[0] != int(manifest["num_codebooks"])
+ or codes.shape[1] == 0
+ ):
+ raise RuntimeError(f"encoder returned invalid codes: {codes.shape}")
+
+ self.voices_root.mkdir(parents=True, exist_ok=True)
+ temp = Path(tempfile.mkdtemp(prefix=f".{voice_name}.", dir=self.voices_root))
+ try:
+ np.save(temp / "codes.npy", codes.astype(np.uint16))
+ meta = {
+ "name": voice_name,
+ "reference_text": reference_text,
+ "shape": list(codes.shape),
+ "dtype": "uint16",
+ "sample_rate": int(manifest["sample_rate"]),
+ "source_audio": Path(filename or "reference_audio").name,
+ "source_sample_rate": source_rate,
+ "source_sha256": hashlib.sha256(data).hexdigest(),
+ "model_fingerprint": manifest["model_fingerprint"],
+ "created_at": datetime.now(timezone.utc).isoformat(),
+ "source_kind": "web_registration",
+ }
+ (temp / "meta.json").write_text(json.dumps(meta, ensure_ascii=False, indent=2) + "\n")
+ if target.exists():
+ shutil.rmtree(target)
+ os.replace(temp, target)
+ finally:
+ if temp.exists():
+ shutil.rmtree(temp)
+ return meta
diff --git a/onnx_runtime/arktts_runtime/runtime.py b/onnx_runtime/arktts_runtime/runtime.py
new file mode 100644
index 0000000..6cfa6ca
--- /dev/null
+++ b/onnx_runtime/arktts_runtime/runtime.py
@@ -0,0 +1,281 @@
+from __future__ import annotations
+
+import json
+import threading
+from collections.abc import Iterator
+from pathlib import Path
+
+import numpy as np
+import onnxruntime as ort
+
+from .prompt import PromptBuilder
+from .voices import VoiceStore
+
+ORT_DTYPES = {
+ "tensor(float)": np.float32,
+ "tensor(float16)": np.float16,
+ "tensor(int64)": np.int64,
+ "tensor(bool)": np.bool_,
+}
+
+
+def _session(path: Path, threads: int | None = None) -> ort.InferenceSession:
+ options = ort.SessionOptions()
+ options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
+ options.log_severity_level = 3
+ if threads is not None:
+ options.intra_op_num_threads = int(threads)
+ options.inter_op_num_threads = max(1, int(threads) // 2)
+ return ort.InferenceSession(str(path), sess_options=options, providers=["CPUExecutionProvider"])
+
+
+def _sample(logits: np.ndarray, temperature: float, top_p: float, top_k: int, rng) -> int:
+ values = np.asarray(logits, dtype=np.float64).reshape(-1)
+ order = np.argsort(values)[::-1]
+ sorted_values = values[order]
+ base = np.exp(sorted_values - np.max(sorted_values))
+ base /= base.sum()
+ cumulative = np.cumsum(base)
+ remove = (cumulative > float(top_p)) | (np.arange(base.size) >= int(top_k))
+ remove[0] = False
+ masked = values.copy()
+ masked[order[remove]] = -np.inf
+ scaled = masked / max(float(temperature), 1e-5)
+ scaled -= np.max(scaled)
+ probs = np.exp(scaled)
+ probs /= probs.sum()
+ noise = -np.log(np.clip(rng.random(probs.size), 1e-12, 1.0))
+ return int(np.argmax(probs / noise))
+
+
+class ArkTtsRuntime:
+ def __init__(
+ self,
+ model_dir: Path,
+ voices_dir: Path,
+ precision: str | None = None,
+ codec_precision: str | None = None,
+ threads: int | None = None,
+ ):
+ self.model_dir = Path(model_dir).resolve()
+ self.manifest = json.loads((self.model_dir / "runtime_manifest.json").read_text())
+ self.precision = precision or self.manifest["default_precision"]
+ if self.precision not in self.manifest["available_precisions"]:
+ raise ValueError(f"unsupported precision: {self.precision}")
+ self.codec_precision = codec_precision or self.manifest.get(
+ "default_codec_precision", "fp16"
+ )
+ available_codec = self.manifest.get("available_codec_precisions", ["fp16"])
+ if self.codec_precision not in available_codec:
+ raise ValueError(f"unsupported codec precision: {self.codec_precision}")
+ self.slow = _session(
+ self.model_dir / f"slow_ar_{self.precision}.onnx",
+ threads,
+ )
+ self.fast = _session(
+ self.model_dir / f"fast_ar_{self.precision}.onnx",
+ threads,
+ )
+ codec_models = self.manifest.get("codec_models", {"fp16": "codec_decoder_fp16.onnx"})
+ self.decoder = _session(
+ self.model_dir / codec_models[self.codec_precision],
+ threads,
+ )
+ self.prompt_builder = PromptBuilder(
+ self.model_dir / "tokenizer",
+ self.manifest["semantic_begin_id"],
+ self.manifest["num_codebooks"],
+ )
+ self.voices = VoiceStore(Path(voices_dir), self.manifest["num_codebooks"])
+ self.slow_inputs = {item.name: item for item in self.slow.get_inputs()}
+ self.fast_inputs = {item.name: item for item in self.fast.get_inputs()}
+
+ def _empty_slow_caches(self) -> list[np.ndarray]:
+ dtype = ORT_DTYPES[self.slow_inputs["cache_key_0"].type]
+ shape = (
+ 1,
+ int(self.manifest["n_local_heads"]),
+ int(self.manifest["max_seq_len"]),
+ int(self.manifest["head_dim"]),
+ )
+ return [np.zeros(shape, dtype=dtype) for _ in range(2 * int(self.manifest["num_layers"]))]
+
+ def _empty_fast_caches(self) -> list[np.ndarray]:
+ dtype = ORT_DTYPES[self.fast_inputs["cache_key_0"].type]
+ shape = (
+ 1,
+ int(self.manifest["fast_n_local_heads"]),
+ int(self.manifest["num_codebooks"]),
+ int(self.manifest["fast_head_dim"]),
+ )
+ return [
+ np.zeros(shape, dtype=dtype) for _ in range(2 * int(self.manifest["num_fast_layers"]))
+ ]
+
+ @staticmethod
+ def _update_caches(
+ caches: list[np.ndarray], positions: np.ndarray, deltas: list[np.ndarray]
+ ) -> None:
+ for index, delta in enumerate(deltas):
+ caches[index][:, :, positions, :] = delta
+
+ def _slow_step(self, codes: np.ndarray, positions: np.ndarray, caches):
+ feeds = {"codes": codes.astype(np.int64), "input_pos": positions.astype(np.int64)}
+ for i in range(int(self.manifest["num_layers"])):
+ feeds[f"cache_key_{i}"] = caches[2 * i]
+ feeds[f"cache_value_{i}"] = caches[2 * i + 1]
+ outputs = self.slow.run(None, feeds)
+ self._update_caches(caches, positions, outputs[2:])
+ return np.asarray(outputs[0])[0, -1], np.asarray(outputs[1])[:, -1:, :]
+
+ def _fast_step(self, hidden, token_id: int, use_hidden: bool, position: int, caches):
+ hidden_dtype = ORT_DTYPES[self.fast_inputs["slow_hidden"].type]
+ feeds = {
+ "slow_hidden": np.asarray(hidden, dtype=hidden_dtype),
+ "token_id": np.asarray([[token_id]], dtype=np.int64),
+ "use_slow_hidden": np.asarray([use_hidden], dtype=np.bool_),
+ "input_pos": np.asarray([position], dtype=np.int64),
+ }
+ for i in range(int(self.manifest["num_fast_layers"])):
+ feeds[f"cache_key_{i}"] = caches[2 * i]
+ feeds[f"cache_value_{i}"] = caches[2 * i + 1]
+ outputs = self.fast.run(None, feeds)
+ self._update_caches(caches, np.asarray([position]), outputs[1:])
+ return np.asarray(outputs[0])[0, -1]
+
+ def _sample_semantic(self, logits, previous, temperature, top_p, top_k, rng):
+ begin = int(self.manifest["semantic_begin_id"])
+ end = int(self.manifest["semantic_end_id"])
+ stop = int(self.manifest["im_end_id"])
+ allowed_ids = np.concatenate([np.arange(begin, end + 1), np.asarray([stop])])
+ values = np.asarray(logits).reshape(-1)
+ allowed_logits = (
+ values
+ if self.manifest.get("slow_logits_layout") == "semantic_then_eos"
+ else values[allowed_ids]
+ )
+ if allowed_logits.size != allowed_ids.size:
+ raise ValueError(
+ f"unexpected slow logits size: {allowed_logits.size}, expected {allowed_ids.size}"
+ )
+ normal_index = _sample(allowed_logits, temperature, top_p, top_k, rng)
+ normal = int(allowed_ids[normal_index])
+ high_index = _sample(allowed_logits, 1.0, 0.9, top_k, rng)
+ high = int(allowed_ids[high_index])
+ if begin <= normal <= end and normal in previous:
+ return high
+ return normal
+
+ def iter_codes(
+ self,
+ text: str,
+ voice: str,
+ max_new_tokens: int = 1024,
+ temperature: float = 0.7,
+ top_p: float = 0.9,
+ top_k: int = 50,
+ seed: int = 42,
+ stop_event: threading.Event | None = None,
+ ) -> Iterator[np.ndarray]:
+ reference_codes, meta = self.voices.load(voice)
+ prompt = self.prompt_builder.build(text, meta["reference_text"], reference_codes)
+ prompt_len = int(prompt.shape[2])
+ max_seq_len = int(self.manifest["max_seq_len"])
+ if prompt_len >= max_seq_len:
+ raise ValueError(
+ f"prompt length {prompt_len} exceeds max sequence length {max_seq_len}"
+ )
+ max_new_tokens = min(int(max_new_tokens), max_seq_len - prompt_len)
+ rng = np.random.default_rng(int(seed))
+ slow_caches = self._empty_slow_caches()
+ positions = np.arange(prompt_len, dtype=np.int64)
+ logits, hidden = self._slow_step(prompt, positions, slow_caches)
+ previous: list[int] = []
+ begin = int(self.manifest["semantic_begin_id"])
+ stop = int(self.manifest["im_end_id"])
+ codebook_size = int(self.manifest["codebook_size"])
+
+ for step in range(max_new_tokens):
+ if stop_event is not None and stop_event.is_set():
+ return
+ semantic = self._sample_semantic(logits, previous, temperature, top_p, top_k, rng)
+ if semantic == stop:
+ return
+ previous.append(semantic)
+ previous = previous[-10:]
+ fast_caches = self._empty_fast_caches()
+ self._fast_step(hidden, 0, True, 0, fast_caches)
+ token = min(max(semantic - begin, 0), codebook_size - 1)
+ codebooks = [token]
+ for fast_pos in range(1, int(self.manifest["num_codebooks"])):
+ fast_logits = self._fast_step(hidden, token, False, fast_pos, fast_caches)
+ token = _sample(fast_logits, temperature, top_p, top_k, rng)
+ codebooks.append(token)
+ frame = np.asarray(codebooks, dtype=np.int64)
+ yield frame
+ if step + 1 >= max_new_tokens:
+ return
+ column = np.concatenate([[semantic], frame]).reshape(1, -1, 1)
+ position = np.asarray([prompt_len + step], dtype=np.int64)
+ logits, hidden = self._slow_step(column, position, slow_caches)
+
+ def decode_codes(self, codes: np.ndarray) -> np.ndarray:
+ values = np.asarray(codes, dtype=np.int64)
+ if values.ndim == 2:
+ values = values[np.newaxis]
+ if values.ndim != 3 or values.shape[1] != int(self.manifest["num_codebooks"]):
+ raise ValueError(f"invalid generated codes shape: {values.shape}")
+ audio = self.decoder.run(None, {"codes": values})[0]
+ return np.asarray(audio, dtype=np.float32).reshape(-1)
+
+ def synthesize(self, **kwargs) -> tuple[np.ndarray, np.ndarray]:
+ frames = list(self.iter_codes(**kwargs))
+ if not frames:
+ raise RuntimeError("model produced no codec frames")
+ codes = np.stack(frames, axis=1)
+ return self.decode_codes(codes), codes
+
+ def stream(self, chunk_frames: int = 12, **kwargs):
+ all_frames: list[np.ndarray] = []
+ emitted_samples = 0
+ hop = int(self.manifest["codec_hop_length"])
+ context = int(self.manifest.get("stream_context_frames", 128))
+ guard = int(self.manifest.get("stream_guard_frames", 1)) * hop
+ seq = 0
+ for frame in self.iter_codes(**kwargs):
+ all_frames.append(frame)
+ if len(all_frames) % int(chunk_frames) != 0:
+ continue
+ start_frame = max(0, len(all_frames) - context - int(chunk_frames))
+ window = np.stack(all_frames[start_frame:], axis=1)
+ audio = self.decode_codes(window)
+ absolute_start = start_frame * hop
+ stable_end = absolute_start + max(0, audio.size - guard)
+ begin = max(0, emitted_samples - absolute_start)
+ end = max(begin, stable_end - absolute_start)
+ if end > begin:
+ chunk = np.ascontiguousarray(audio[begin:end])
+ emitted_samples += chunk.size
+ yield {
+ "type": "audio_chunk",
+ "seq": seq,
+ "audio": chunk,
+ "frame_count": len(all_frames),
+ }
+ seq += 1
+ if not all_frames:
+ raise RuntimeError("model produced no codec frames")
+ start_frame = max(0, len(all_frames) - context - int(chunk_frames))
+ window = np.stack(all_frames[start_frame:], axis=1)
+ audio = self.decode_codes(window)
+ absolute_start = start_frame * hop
+ begin = max(0, emitted_samples - absolute_start)
+ if begin < audio.size:
+ chunk = np.ascontiguousarray(audio[begin:])
+ yield {
+ "type": "audio_chunk",
+ "seq": seq,
+ "audio": chunk,
+ "frame_count": len(all_frames),
+ }
+ yield {"type": "complete", "codes": np.stack(all_frames, axis=1)}
diff --git a/onnx_runtime/arktts_runtime/service.py b/onnx_runtime/arktts_runtime/service.py
new file mode 100644
index 0000000..5d4d253
--- /dev/null
+++ b/onnx_runtime/arktts_runtime/service.py
@@ -0,0 +1,614 @@
+from __future__ import annotations
+
+# ruff: noqa: B008, E501
+import base64
+import ctypes
+import gc
+import json
+import os
+import re
+import subprocess
+import threading
+import time
+from pathlib import Path
+
+import numpy as np
+import soundfile as sf
+from fastapi import FastAPI, File, Form, HTTPException, UploadFile
+from fastapi.responses import HTMLResponse, Response, StreamingResponse
+from pydantic import BaseModel, Field
+
+from .registration import MAX_AUDIO_BYTES, VoiceRegistration
+from .runtime import ArkTtsRuntime
+
+ROOT = Path(__file__).resolve().parents[1]
+MODEL_DIR = Path(os.environ.get("ARKTTS_MODEL_DIR", ROOT / "model")).resolve()
+VOICES_DIR = Path(os.environ.get("ARKTTS_VOICES_DIR", ROOT / "voices")).resolve()
+PRECISION = os.environ.get("ARKTTS_PRECISION")
+CODEC_PRECISION = os.environ.get("ARKTTS_CODEC_PRECISION")
+THREADS = int(os.environ.get("ARKTTS_THREADS", "5"))
+REGISTRATION_DIR = Path(
+ os.environ.get(
+ "ARKTTS_REGISTRATION_DIR",
+ MODEL_DIR / "registration",
+ )
+).resolve()
+runtime: ArkTtsRuntime | None = None
+registration: VoiceRegistration | None = None
+request_lock = threading.Lock()
+registration_lock = threading.Lock()
+reload_lock = threading.Lock()
+active_stop: threading.Event | None = None
+service_started_at = time.monotonic()
+memory_lock = threading.Lock()
+memory_cache: dict[str, float | None] = {"sampled_at": 0.0, "current_mb": None, "peak_mb": None}
+app = FastAPI(title="Audio8 TTS ONNX Runtime")
+
+
+TEST_PAGE = """<!doctype html>
+<html lang="zh-CN">
+<head>
+ <meta charset="utf-8">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Audio8 TTS 本地测试</title>
+ <style>
+ :root { color-scheme: light; --ink:#18201f; --muted:#65706e; --line:#d7dddb; --paper:#f5f7f6; --accent:#087f5b; --accent-dark:#066447; --danger:#b42318; }
+ * { box-sizing:border-box; }
+ body { margin:0; min-width:320px; background:var(--paper); color:var(--ink); font:15px/1.5 -apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif; letter-spacing:0; }
+ header { border-bottom:1px solid var(--line); background:#fff; }
+ .bar, main { width:min(880px, calc(100% - 32px)); margin:auto; }
+ .bar { min-height:64px; display:flex; align-items:center; justify-content:space-between; gap:16px; }
+ h1 { margin:0; font-size:20px; font-weight:680; }
+ .header-actions, .health { display:flex; align-items:center; gap:8px; }
+ .health { color:var(--muted); font-size:13px; }
+ .icon-button { width:36px; height:36px; padding:0; border:1px solid var(--line); border-radius:6px; background:#fff; color:var(--ink); font:22px/34px -apple-system,BlinkMacSystemFont,sans-serif; }
+ .icon-button:hover { border-color:#aeb8b5; background:#f7f9f8; }
+ .icon-button:disabled { opacity:.45; }
+ .dot { width:9px; height:9px; border-radius:50%; background:#a3aaa8; }
+ .dot.ok { background:#12a371; }
+ main { padding:32px 0 48px; }
+ .meta { display:flex; flex-wrap:wrap; gap:8px 18px; margin-bottom:18px; color:var(--muted); font-size:13px; }
+ .metrics { display:grid; grid-template-columns:repeat(3, 1fr); border-top:1px solid var(--line); border-bottom:1px solid var(--line); margin-bottom:24px; }
+ .metric { min-width:0; padding:15px 0; }
+ .metric + .metric { padding-left:18px; border-left:1px solid var(--line); }
+ .metric span { display:block; color:var(--muted); font-size:12px; }
+ .metric strong { display:block; margin-top:2px; font-size:19px; font-weight:680; font-variant-numeric:tabular-nums; }
+ .memory-track { grid-column:1 / -1; height:3px; margin-top:-3px; overflow:hidden; background:#dfe5e3; }
+ .memory-fill { width:0; height:100%; background:var(--accent); transition:width .35s ease; }
+ .tabs { display:flex; width:100%; margin-bottom:24px; border-bottom:1px solid var(--line); }
+ .tab { height:42px; padding:0 18px; border:0; border-bottom:2px solid transparent; border-radius:0; background:transparent; color:var(--muted); }
+ .tab:hover { background:transparent; color:var(--ink); }
+ .tab.active { border-bottom-color:var(--accent); color:var(--ink); }
+ .view[hidden] { display:none; }
+ label { display:block; margin:0 0 7px; font-size:13px; font-weight:650; }
+ textarea, select, input[type=number], input[type=text], input[type=file] { width:100%; border:1px solid #bfc8c5; border-radius:6px; background:#fff; color:var(--ink); font:inherit; outline:none; }
+ textarea:focus, select:focus, input:focus { border-color:var(--accent); box-shadow:0 0 0 3px rgba(8,127,91,.12); }
+ textarea { min-height:156px; resize:vertical; padding:13px 14px; }
+ select, input[type=number], input[type=text], input[type=file] { height:42px; padding:0 11px; }
+ input[type=file] { padding:8px; }
+ .row { display:grid; grid-template-columns:1.35fr repeat(3, 1fr); gap:14px; margin-top:18px; }
+ .actions { display:flex; align-items:center; gap:10px; margin-top:24px; }
+ button, .download { height:42px; border-radius:6px; padding:0 17px; font:650 14px/40px inherit; cursor:pointer; text-decoration:none; }
+ button { border:1px solid var(--accent); background:var(--accent); color:#fff; }
+ button:hover { background:var(--accent-dark); }
+ button:disabled { cursor:not-allowed; opacity:.55; }
+ .download { display:none; border:1px solid #bfc8c5; background:#fff; color:var(--ink); }
+ .status { min-height:22px; margin:18px 0 10px; color:var(--muted); font-size:13px; }
+ .status.error { color:var(--danger); }
+ audio { display:none; width:100%; margin-top:10px; }
+ audio.ready { display:block; }
+ .register-grid { display:grid; grid-template-columns:1fr 1fr; gap:14px; margin-bottom:18px; }
+ .register-text { min-height:112px; }
+ .check { display:flex; align-items:center; gap:8px; margin:14px 0 0; font-weight:500; }
+ .check input { width:16px; height:16px; accent-color:var(--accent); }
+ .audio-preview { display:none; margin:14px 0 0; }
+ .audio-preview.ready { display:block; }
+ @media (max-width:700px) { .row { grid-template-columns:1fr 1fr; } main { padding-top:24px; } }
+ @media (max-width:440px) { .row, .register-grid { grid-template-columns:1fr; } .metrics { grid-template-columns:1fr 1fr; } .metric:nth-child(3) { grid-column:1 / -1; padding-left:0; border-left:0; border-top:1px solid var(--line); } .actions { align-items:stretch; flex-direction:column; } button, .download { width:100%; text-align:center; } }
+ </style>
+</head>
+<body>
+ <header><div class="bar"><h1>Audio8 TTS</h1><div class="header-actions"><div class="health"><span id="dot" class="dot"></span><span id="health">正在加载模型</span></div><button id="reloadRuntime" class="icon-button" type="button" title="重新加载模型" aria-label="重新加载模型">↻</button></div></div></header>
+ <main>
+ <div class="meta"><span id="precision">DualAR:-</span><span id="codecPrecision">Codec:-</span><span id="provider">执行器:-</span></div>
+ <section class="metrics" aria-label="运行状态">
+ <div class="metric"><span>当前内存</span><strong id="memoryCurrent">读取中</strong></div>
+ <div class="metric"><span>峰值内存</span><strong id="memoryPeak">读取中</strong></div>
+ <div class="metric"><span>运行时间</span><strong id="uptime">-</strong></div>
+ <div class="memory-track"><div id="memoryFill" class="memory-fill"></div></div>
+ </section>
+ <div class="tabs" role="tablist"><button id="ttsTab" class="tab active" role="tab">语音合成</button><button id="registerTab" class="tab" role="tab">音频注册</button></div>
+ <section id="ttsView" class="view">
+ <label for="text">合成文本</label>
+ <textarea id="text">你好,这是 Audio8 TTS 的本地语音合成测试。</textarea>
+ <div class="row">
+ <div><label for="voice">音色</label><select id="voice"></select></div>
+ <div><label for="temperature">Temperature</label><input id="temperature" type="number" value="0.3" min="0.05" max="2" step="0.05"></div>
+ <div><label for="topP">Top P</label><input id="topP" type="number" value="0.9" min="0.05" max="1" step="0.05"></div>
+ <div><label for="seed">Seed</label><input id="seed" type="number" value="42" min="0" step="1"></div>
+ </div>
+ <div class="actions"><button id="generate" disabled>生成语音</button><a id="download" class="download" download="audio8_output.wav">下载 WAV</a></div>
+ <div id="status" class="status"></div>
+ <audio id="audio" controls></audio>
+ </section>
+ <section id="registerView" class="view" hidden>
+ <div class="register-grid">
+ <div><label for="referenceAudio">参考音频</label><input id="referenceAudio" type="file" accept="audio/*,.wav,.flac,.mp3,.m4a"></div>
+ <div><label for="voiceName">音色名称</label><input id="voiceName" type="text" maxlength="64" placeholder="speaker_a"></div>
+ </div>
+ <label for="referenceText">参考音频原文</label>
+ <textarea id="referenceText" class="register-text"></textarea>
+ <audio id="referencePreview" class="audio-preview" controls></audio>
+ <label class="check"><input id="overwriteVoice" type="checkbox">覆盖同名音色</label>
+ <div class="actions"><button id="registerVoice" disabled>注册音色</button></div>
+ <div id="registerStatus" class="status"></div>
+ </section>
+ </main>
+ <script>
+ const $ = id => document.getElementById(id);
+ let audioUrl = null;
+ let referenceUrl = null;
+ let registrationAvailable = false;
+ function selectView(name) {
+ const register = name === 'register';
+ $('ttsView').hidden = register;
+ $('registerView').hidden = !register;
+ $('ttsTab').classList.toggle('active', !register);
+ $('registerTab').classList.toggle('active', register);
+ }
+ async function refreshVoices(selected) {
+ const response = await fetch('/api/voices');
+ if (!response.ok) throw new Error('读取音色失败');
+ const data = await response.json();
+ const names = data.voices.map(voice => typeof voice === 'string' ? voice : voice.name);
+ $('voice').replaceChildren(...names.map(name => new Option(name, name)));
+ if (selected && names.includes(selected)) $('voice').value = selected;
+ $('generate').disabled = names.length === 0;
+ return names;
+ }
+ function formatUptime(seconds) {
+ const value = Math.max(0, Math.floor(seconds));
+ const hours = Math.floor(value / 3600);
+ const minutes = Math.floor((value % 3600) / 60);
+ const secs = value % 60;
+ return hours ? `${hours}时 ${minutes}分 ${secs}秒` : `${minutes}分 ${secs}秒`;
+ }
+ async function refreshMetrics() {
+ try {
+ const response = await fetch('/api/system', {cache:'no-store'});
+ if (!response.ok) throw new Error(`HTTP ${response.status}`);
+ const data = await response.json();
+ $('memoryCurrent').textContent = data.memory.current_mb == null ? '不可用' : `${(data.memory.current_mb / 1024).toFixed(2)} GB`;
+ $('memoryPeak').textContent = data.memory.peak_mb == null ? '不可用' : `${(data.memory.peak_mb / 1024).toFixed(2)} GB`;
+ $('uptime').textContent = formatUptime(data.uptime_seconds);
+ const scale = Math.max(4096, data.memory.peak_mb || 0);
+ $('memoryFill').style.width = `${Math.min(100, ((data.memory.current_mb || 0) / scale) * 100)}%`;
+ } catch (_) {
+ $('memoryCurrent').textContent = '读取失败';
+ }
+ }
+ async function initialize() {
+ try {
+ const [health, registrationStatus] = await Promise.all([fetch('/api/health'), fetch('/api/registration/status')]);
+ if (!health.ok || !registrationStatus.ok) throw new Error('服务初始化失败');
+ const healthData = await health.json();
+ const registrationData = await registrationStatus.json();
+ $('precision').textContent = `DualAR:${healthData.precision.toUpperCase()}`;
+ $('codecPrecision').textContent = `Codec:${healthData.codec_precision.toUpperCase()}`;
+ $('provider').textContent = `执行器:${healthData.providers.slow[0]}`;
+ const voiceNames = await refreshVoices();
+ registrationAvailable = registrationData.available;
+ $('registerVoice').disabled = !registrationAvailable;
+ $('registerStatus').textContent = registrationAvailable ? '' : registrationData.reason;
+ $('dot').classList.add('ok');
+ $('health').textContent = '模型已就绪';
+ $('generate').disabled = voiceNames.length === 0;
+ $('status').textContent = voiceNames.length ? '' : '没有可用音色';
+ if (voiceNames.length === 0 && registrationAvailable) selectView('register');
+ } catch (error) {
+ $('health').textContent = '服务不可用';
+ $('status').textContent = error.message;
+ $('status').classList.add('error');
+ }
+ }
+ $('generate').addEventListener('click', async () => {
+ const text = $('text').value.trim();
+ if (!text) { $('status').textContent = '请输入合成文本'; return; }
+ $('generate').disabled = true;
+ $('status').classList.remove('error');
+ $('status').textContent = '正在生成语音...';
+ $('audio').classList.remove('ready');
+ $('download').style.display = 'none';
+ const started = performance.now();
+ try {
+ const response = await fetch('/api/tts', {
+ method:'POST', headers:{'Content-Type':'application/json'},
+ body:JSON.stringify({text, voice_name:$('voice').value, temperature:Number($('temperature').value), top_p:Number($('topP').value), seed:Number($('seed').value)})
+ });
+ if (!response.ok) throw new Error((await response.text()) || `HTTP ${response.status}`);
+ const blob = await response.blob();
+ if (audioUrl) URL.revokeObjectURL(audioUrl);
+ audioUrl = URL.createObjectURL(blob);
+ $('audio').src = audioUrl;
+ $('audio').classList.add('ready');
+ $('download').href = audioUrl;
+ $('download').style.display = 'inline-block';
+ $('status').textContent = `生成完成,用时 ${((performance.now()-started)/1000).toFixed(2)} 秒`;
+ await $('audio').play().catch(() => {});
+ } catch (error) {
+ $('status').textContent = `生成失败:${error.message}`;
+ $('status').classList.add('error');
+ } finally { $('generate').disabled = false; }
+ });
+ $('ttsTab').addEventListener('click', () => selectView('tts'));
+ $('registerTab').addEventListener('click', () => selectView('register'));
+ $('referenceAudio').addEventListener('change', () => {
+ const file = $('referenceAudio').files[0];
+ if (referenceUrl) URL.revokeObjectURL(referenceUrl);
+ if (!file) { $('referencePreview').classList.remove('ready'); return; }
+ referenceUrl = URL.createObjectURL(file);
+ $('referencePreview').src = referenceUrl;
+ $('referencePreview').classList.add('ready');
+ });
+ $('registerVoice').addEventListener('click', async () => {
+ const file = $('referenceAudio').files[0];
+ const name = $('voiceName').value.trim();
+ const text = $('referenceText').value.trim();
+ if (!file || !name || !text) { $('registerStatus').textContent = '请选择音频并填写音色名称和准确原文'; return; }
+ $('registerVoice').disabled = true;
+ $('registerStatus').classList.remove('error');
+ $('registerStatus').textContent = '正在注册音色...';
+ const form = new FormData();
+ form.append('audio', file);
+ form.append('name', name);
+ form.append('text', text);
+ form.append('overwrite', $('overwriteVoice').checked ? 'true' : 'false');
+ try {
+ const response = await fetch('/api/voices/register', {method:'POST', body:form});
+ if (!response.ok) throw new Error((await response.json()).detail || `HTTP ${response.status}`);
+ const result = await response.json();
+ await refreshVoices(result.voice.name);
+ $('registerStatus').textContent = `注册完成:${result.voice.name}(${result.voice.shape[1]} 帧)`;
+ } catch (error) {
+ $('registerStatus').textContent = `注册失败:${error.message}`;
+ $('registerStatus').classList.add('error');
+ } finally { $('registerVoice').disabled = !registrationAvailable; }
+ });
+ $('reloadRuntime').addEventListener('click', async () => {
+ const button = $('reloadRuntime');
+ button.disabled = true;
+ $('dot').classList.remove('ok');
+ $('health').textContent = '正在重新加载模型';
+ try {
+ const response = await fetch('/api/runtime/reload', {method:'POST'});
+ const result = await response.json();
+ if (!response.ok) throw new Error(result.detail || `HTTP ${response.status}`);
+ $('precision').textContent = `DualAR:${result.precision.toUpperCase()}`;
+ $('codecPrecision').textContent = `Codec:${result.codec_precision.toUpperCase()}`;
+ await refreshVoices();
+ await refreshMetrics();
+ $('dot').classList.add('ok');
+ $('health').textContent = `模型已重新加载(${result.reload_seconds.toFixed(2)} 秒)`;
+ } catch (error) {
+ $('health').textContent = `重新加载失败:${error.message}`;
+ } finally { button.disabled = false; }
+ });
+ initialize();
+ refreshMetrics();
+ setInterval(refreshMetrics, 2000);
+ </script>
+</body>
+</html>"""
+
+
+class TtsRequest(BaseModel):
+ text: str = Field(..., min_length=1, max_length=1000)
+ voice_name: str
+ max_new_tokens: int = Field(1024, ge=16, le=2048)
+ temperature: float = Field(0.3, gt=0.0, le=2.0)
+ top_p: float = Field(0.9, gt=0.0, le=1.0)
+ top_k: int = Field(50, ge=1, le=4096)
+ seed: int = 42
+
+
+class OpenAiSpeechRequest(BaseModel):
+ model: str = "arktts"
+ input: str = Field(..., min_length=1, max_length=1000)
+ voice: str
+ response_format: str = "wav"
+
+
[email protected]("/", response_class=HTMLResponse)
+def test_page():
+ return TEST_PAGE
+
+
[email protected]_event("startup")
+def startup() -> None:
+ global registration, runtime
+ runtime = ArkTtsRuntime(MODEL_DIR, VOICES_DIR, PRECISION, CODEC_PRECISION, THREADS)
+ registration = VoiceRegistration(
+ REGISTRATION_DIR,
+ runtime.voices.root,
+ str(runtime.manifest["model_fingerprint"]),
+ )
+
+
+def require_runtime() -> ArkTtsRuntime:
+ if runtime is None:
+ raise HTTPException(503, "runtime is not loaded")
+ return runtime
+
+
+def require_registration() -> VoiceRegistration:
+ if registration is None:
+ raise HTTPException(503, "registration is not initialized")
+ return registration
+
+
+def release_allocator_memory() -> None:
+ gc.collect()
+ if os.uname().sysname != "Darwin":
+ return
+ try:
+ system = ctypes.CDLL("/usr/lib/libSystem.B.dylib")
+ system.malloc_zone_pressure_relief.argtypes = [ctypes.c_void_p, ctypes.c_size_t]
+ system.malloc_zone_pressure_relief.restype = ctypes.c_size_t
+ system.malloc_zone_pressure_relief(None, 0)
+ except (AttributeError, OSError):
+ pass
+
+
+def clear_runtime_cache() -> None:
+ with memory_lock:
+ memory_cache.update(sampled_at=0.0, current_mb=None, peak_mb=None)
+
+
+def _memory_value_mb(output: str, pattern: str) -> float | None:
+ match = re.search(pattern, output, flags=re.IGNORECASE)
+ if match is None:
+ return None
+ value = float(match.group(1))
+ unit = match.group(2).upper()
+ return value * {"KB": 1 / 1024, "MB": 1, "GB": 1024}[unit]
+
+
+def memory_stats() -> dict[str, float | None]:
+ now = time.monotonic()
+ with memory_lock:
+ if now - float(memory_cache["sampled_at"] or 0.0) < 1.0:
+ return memory_cache.copy()
+ try:
+ result = subprocess.run(
+ ["/usr/bin/footprint", str(os.getpid())],
+ capture_output=True,
+ check=True,
+ text=True,
+ timeout=5,
+ )
+ current_mb = _memory_value_mb(result.stdout, r"Footprint:\s*([\d.]+)\s*(KB|MB|GB)")
+ peak_mb = _memory_value_mb(
+ result.stdout, r"phys_footprint_peak:\s*([\d.]+)\s*(KB|MB|GB)"
+ )
+ except (OSError, subprocess.SubprocessError):
+ current_mb = None
+ peak_mb = None
+ memory_cache.update(sampled_at=now, current_mb=current_mb, peak_mb=peak_mb)
+ return memory_cache.copy()
+
+
[email protected]("/api/health")
+def health():
+ obj = require_runtime()
+ return {
+ "ok": True,
+ "precision": obj.precision,
+ "codec_precision": obj.codec_precision,
+ "providers": {
+ "slow": obj.slow.get_providers(),
+ "fast": obj.fast.get_providers(),
+ "decoder": obj.decoder.get_providers(),
+ },
+ }
+
+
[email protected]("/api/voices")
+def voices():
+ return {"voices": require_runtime().voices.list()}
+
+
[email protected]("/api/registration/status")
+def registration_status():
+ return require_registration().status()
+
+
[email protected]("/api/voices/register")
+def register_voice(
+ audio: UploadFile = File(...),
+ text: str = Form(...),
+ name: str = Form(...),
+ overwrite: bool = Form(False),
+):
+ global runtime
+ data = audio.file.read(MAX_AUDIO_BYTES + 1)
+ try:
+ with registration_lock, request_lock:
+ current = require_runtime()
+ precision = current.precision
+ codec_precision = current.codec_precision
+ runtime = None
+ del current
+ clear_runtime_cache()
+ release_allocator_memory()
+ try:
+ meta = require_registration().register(
+ data,
+ audio.filename or "reference_audio",
+ text,
+ name,
+ overwrite,
+ )
+ finally:
+ release_allocator_memory()
+ runtime = ArkTtsRuntime(MODEL_DIR, VOICES_DIR, precision, codec_precision, THREADS)
+ clear_runtime_cache()
+ except FileExistsError as exc:
+ raise HTTPException(409, str(exc)) from exc
+ except ValueError as exc:
+ raise HTTPException(400, str(exc)) from exc
+ except RuntimeError as exc:
+ raise HTTPException(503, str(exc)) from exc
+ return {"ok": True, "voice": meta}
+
+
[email protected]("/api/system")
+def system_stats():
+ stats = memory_stats()
+ return {
+ "memory": {"current_mb": stats["current_mb"], "peak_mb": stats["peak_mb"]},
+ "uptime_seconds": time.monotonic() - service_started_at,
+ "refresh_interval_ms": 2000,
+ }
+
+
[email protected]("/api/runtime/reload")
+def reload_runtime():
+ global active_stop, registration, runtime
+ if active_stop is not None:
+ active_stop.set()
+ started = time.monotonic()
+ with reload_lock, request_lock:
+ old_runtime = runtime
+ precision = old_runtime.precision if old_runtime is not None else PRECISION
+ codec_precision = (
+ old_runtime.codec_precision if old_runtime is not None else CODEC_PRECISION
+ )
+ runtime = None
+ registration = None
+ del old_runtime
+ clear_runtime_cache()
+ release_allocator_memory()
+ try:
+ runtime = ArkTtsRuntime(MODEL_DIR, VOICES_DIR, precision, codec_precision, THREADS)
+ registration = VoiceRegistration(
+ REGISTRATION_DIR,
+ runtime.voices.root,
+ str(runtime.manifest["model_fingerprint"]),
+ )
+ except Exception as exc:
+ raise HTTPException(500, f"failed to reload runtime: {exc}") from exc
+ return {
+ "ok": True,
+ "precision": runtime.precision,
+ "codec_precision": runtime.codec_precision,
+ "reload_seconds": time.monotonic() - started,
+ }
+
+
[email protected]("/api/tts/stream")
+def stream_tts(request: TtsRequest):
+ def events():
+ global active_stop
+ stop = threading.Event()
+ active_stop = stop
+ try:
+ with request_lock:
+ obj = require_runtime()
+ yield (
+ json.dumps(
+ {
+ "event": "start",
+ "sample_rate": int(obj.manifest["sample_rate"]),
+ "channels": 1,
+ "sample_format": "s16le",
+ "precision": obj.precision,
+ "codec_precision": obj.codec_precision,
+ }
+ )
+ + "\n"
+ )
+ for event in obj.stream(
+ text=request.text,
+ voice=request.voice_name,
+ max_new_tokens=request.max_new_tokens,
+ temperature=request.temperature,
+ top_p=request.top_p,
+ top_k=request.top_k,
+ seed=request.seed,
+ stop_event=stop,
+ ):
+ if stop.is_set():
+ yield json.dumps({"event": "cancelled"}) + "\n"
+ return
+ if event["type"] == "audio_chunk":
+ pcm = (np.clip(event["audio"], -1.0, 1.0) * 32767.0).astype("<i2")
+ yield (
+ json.dumps(
+ {
+ "event": "audio_chunk",
+ "seq": int(event["seq"]),
+ "frame_count": int(event["frame_count"]),
+ "pcm_b64": base64.b64encode(pcm.tobytes()).decode("ascii"),
+ }
+ )
+ + "\n"
+ )
+ else:
+ yield (
+ json.dumps(
+ {"event": "complete", "frame_count": int(event["codes"].shape[1])}
+ )
+ + "\n"
+ )
+ finally:
+ if active_stop is stop:
+ active_stop = None
+
+ return StreamingResponse(events(), media_type="application/x-ndjson")
+
+
[email protected]("/api/tts/cancel")
+def cancel():
+ if active_stop is None:
+ return {"ok": True, "cancelled": False}
+ active_stop.set()
+ return {"ok": True, "cancelled": True}
+
+
[email protected]("/v1/audio/speech")
+def openai_speech(request: OpenAiSpeechRequest):
+ if request.model not in {"arktts", "tts-1"}:
+ raise HTTPException(400, f"unsupported model: {request.model}")
+ if request.response_format not in {"wav", "pcm"}:
+ raise HTTPException(400, "response_format must be wav or pcm")
+ with request_lock:
+ obj = require_runtime()
+ audio, _ = obj.synthesize(text=request.input, voice=request.voice)
+ if request.response_format == "pcm":
+ pcm = (np.clip(audio, -1.0, 1.0) * 32767.0).astype("<i2").tobytes()
+ return Response(pcm, media_type="application/octet-stream")
+ import io
+
+ buffer = io.BytesIO()
+ sf.write(buffer, audio, int(obj.manifest["sample_rate"]), format="WAV")
+ return Response(buffer.getvalue(), media_type="audio/wav")
+
+
[email protected]("/api/tts")
+def tts(request: TtsRequest):
+ with request_lock:
+ obj = require_runtime()
+ audio, _ = obj.synthesize(
+ text=request.text,
+ voice=request.voice_name,
+ max_new_tokens=request.max_new_tokens,
+ temperature=request.temperature,
+ top_p=request.top_p,
+ top_k=request.top_k,
+ seed=request.seed,
+ )
+ import io
+
+ buffer = io.BytesIO()
+ sf.write(buffer, audio, int(obj.manifest["sample_rate"]), format="WAV")
+ return Response(buffer.getvalue(), media_type="audio/wav")
diff --git a/onnx_runtime/arktts_runtime/voices.py b/onnx_runtime/arktts_runtime/voices.py
new file mode 100644
index 0000000..74985c2
--- /dev/null
+++ b/onnx_runtime/arktts_runtime/voices.py
@@ -0,0 +1,40 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import numpy as np
+
+
+class VoiceStore:
+ def __init__(self, root: Path, num_codebooks: int):
+ self.root = root.resolve()
+ self.num_codebooks = int(num_codebooks)
+
+ def list(self) -> list[dict]:
+ voices = []
+ if not self.root.exists():
+ return voices
+ for meta_path in sorted(self.root.glob("*/meta.json")):
+ try:
+ voices.append(json.loads(meta_path.read_text()))
+ except (OSError, json.JSONDecodeError):
+ continue
+ return voices
+
+ def load(self, name: str) -> tuple[np.ndarray, dict]:
+ if not name or Path(name).name != name:
+ raise ValueError("invalid voice name")
+ voice_dir = self.root / name
+ meta_path = voice_dir / "meta.json"
+ codes_path = voice_dir / "codes.npy"
+ if not meta_path.is_file() or not codes_path.is_file():
+ raise KeyError(f"voice not found: {name}")
+ meta = json.loads(meta_path.read_text())
+ codes = np.load(codes_path, allow_pickle=False).astype(np.int64, copy=False)
+ if codes.ndim != 2 or codes.shape[0] != self.num_codebooks or codes.shape[1] == 0:
+ raise ValueError(f"invalid codes for voice {name}: {codes.shape}")
+ reference_text = str(meta.get("reference_text", "")).strip()
+ if not reference_text:
+ raise ValueError(f"voice {name} has no reference_text")
+ return codes, meta
diff --git a/onnx_runtime/requirements.txt b/onnx_runtime/requirements.txt
new file mode 100644
index 0000000..308e373
--- /dev/null
+++ b/onnx_runtime/requirements.txt
@@ -0,0 +1,8 @@
+numpy==2.4.3
+onnxruntime==1.24.4
+soundfile==0.13.1
+scipy==1.17.1
+fastapi==0.135.2
+uvicorn==0.42.0
+pydantic>=2.9,<3
+python-multipart==0.0.22
diff --git a/onnx_runtime/run_infer.sh b/onnx_runtime/run_infer.sh
new file mode 100755
index 0000000..f652811
--- /dev/null
+++ b/onnx_runtime/run_infer.sh
@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")" && pwd)"
+exec "$ROOT/.venv/bin/python" -m arktts_runtime.cli \
+ --model-dir "${ARKTTS_MODEL_DIR:-$ROOT/model}" \
+ --voices-dir "${ARKTTS_VOICES_DIR:-$ROOT/voices}" \
+ "$@"
diff --git a/onnx_runtime/run_server.sh b/onnx_runtime/run_server.sh
new file mode 100755
index 0000000..b8f7eeb
--- /dev/null
+++ b/onnx_runtime/run_server.sh
@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")" && pwd)"
+export ARKTTS_MODEL_DIR="${ARKTTS_MODEL_DIR:-$ROOT/model}"
+export ARKTTS_VOICES_DIR="${ARKTTS_VOICES_DIR:-$ROOT/voices}"
+export ARKTTS_REGISTRATION_DIR="${ARKTTS_REGISTRATION_DIR:-$ARKTTS_MODEL_DIR/registration}"
+export ARKTTS_PRECISION="int4"
+export ARKTTS_CODEC_PRECISION="fp16"
+export ARKTTS_THREADS="${ARKTTS_THREADS:-5}"
+
+exec "$ROOT/.venv/bin/uvicorn" arktts_runtime.service:app \
+ --app-dir "$ROOT" \
+ --host "${HOST:-127.0.0.1}" \
+ --port "${PORT:-8024}"
diff --git a/onnx_runtime/setup.sh b/onnx_runtime/setup.sh
new file mode 100755
index 0000000..4f17fda
--- /dev/null
+++ b/onnx_runtime/setup.sh
@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")" && pwd)"
+PYTHON_BIN="${PYTHON_BIN:-python3}"
+
+if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
+ echo "Python was not found. Set PYTHON_BIN to a Python 3.11+ executable." >&2
+ exit 1
+fi
+
+if ! "$PYTHON_BIN" -c 'import sys; raise SystemExit(sys.version_info < (3, 11))'; then
+ echo "Python 3.11 or newer is required." >&2
+ exit 1
+fi
+
+"$PYTHON_BIN" -m venv "$ROOT/.venv"
+"$ROOT/.venv/bin/python" -m pip install --upgrade pip
+"$ROOT/.venv/bin/python" -m pip install -r "$ROOT/requirements.txt"
+# Only the local Rust tokenizer runtime is needed. Its optional Hub dependency
+# tree (huggingface-hub, httpx, hf-xet, and friends) is not used here.
+"$ROOT/.venv/bin/python" -m pip install --no-deps tokenizers==0.22.2
+echo "Environment ready: $ROOT/.venv"
diff --git a/onnx_runtime/start_server.sh b/onnx_runtime/start_server.sh
new file mode 100755
index 0000000..7d58c60
--- /dev/null
+++ b/onnx_runtime/start_server.sh
@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")" && pwd)"
+PID_FILE="$ROOT/.service.pid"
+PORT="${PORT:-8024}"
+
+if [[ -f "$PID_FILE" ]] && kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then
+ echo "Service is already running with PID $(cat "$PID_FILE")"
+ exit 0
+fi
+
+PORT="$PORT" nohup "$ROOT/run_server.sh" >"$ROOT/service.log" 2>&1 &
+PID=$!
+echo "$PID" >"$PID_FILE"
+
+for _ in {1..60}; do
+ if curl -fsS "http://127.0.0.1:$PORT/api/health" >/dev/null 2>&1; then
+ echo "Audio8 TTS is ready: http://127.0.0.1:$PORT"
+ exit 0
+ fi
+ if ! kill -0 "$PID" 2>/dev/null; then
+ echo "Service exited during startup. See $ROOT/service.log" >&2
+ exit 1
+ fi
+ sleep 1
+done
+
+echo "Service did not become ready in 60 seconds. See $ROOT/service.log" >&2
+exit 1
diff --git a/onnx_runtime/stop_server.sh b/onnx_runtime/stop_server.sh
new file mode 100755
index 0000000..3963bff
--- /dev/null
+++ b/onnx_runtime/stop_server.sh
@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")" && pwd)"
+PID_FILE="$ROOT/.service.pid"
+
+if [[ ! -f "$PID_FILE" ]]; then
+ echo "No managed Audio8 TTS service is running."
+ exit 0
+fi
+
+PID="$(cat "$PID_FILE")"
+if kill -0 "$PID" 2>/dev/null; then
+ kill "$PID"
+ for _ in {1..30}; do
+ if ! kill -0 "$PID" 2>/dev/null; then
+ break
+ fi
+ sleep 1
+ done
+fi
+rm -f "$PID_FILE"
+echo "Audio8 TTS service stopped."