Skip to content

SOTA-Class TTS at Compact Scale

Python

Add standalone 0.1B INT8 ONNX runtime

chuarunyuan4e8e99fe9286
diff --git a/onnx_runtime_0_1b_int8/README.md b/onnx_runtime_0_1b_int8/README.md
new file mode 100644
index 0000000..6836efe
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/README.md
@@ -0,0 +1,99 @@
+# Audio8 0.1B INT8 ONNX Runtime
+
+This directory is a self-contained runtime for the
+`Audio8/audio8-TTS-0.1B-ONNX-INT8` export. It is intentionally separate from
+the existing `onnx_runtime/` implementation, which targets the incompatible
+0.6B INT4 graph.
+
+## Why this runtime is separate
+
+The 0.1B checkpoint uses a Falcon-H1 hybrid slow graph. The graph accepts one
+`[1, 11, 1]` token column and carries four recurrent tensors between calls:
+
+```text
+cache_keys [24, 1, 2, 2048, 64]
+cache_values [24, 1, 2, 2048, 64]
+conv_states [24, 1, 896, 4]
+ssm_states [24, 1, 24, 32, 64]
+```
+
+The implementation below performs prompt prefill one position at a time,
+updates the returned state deltas, maps the compact 4097-way semantic logits,
+then runs the four-layer Fast AR codebook graph and the FP16 codec decoder.
+
+## Install
+
+Python 3.10 or newer is supported, including the Python 3.10 environment used
+by the Jetson issue report. From this directory:
+
+```bash
+python3 -m pip install -U "huggingface_hub[cli]"
+hf download Audio8/audio8-TTS-0.1B-ONNX-INT8 --local-dir model
+bash setup.sh
+python scripts/register_default_voice.py
+```
+
+The model directory must contain:
+
+```text
+model/
+|- slow_ar_int8.onnx(.data)
+|- fast_ar_int8.onnx(.data)
+|- codec_decoder_fp16.onnx(.data)
+|- runtime_manifest.json
+|- reference_codes.npy
+|- tokenizer/tokenizer.json
+`- registration/
+ |- codec_encoder_fp16.onnx(.data) (optional)
+ `- registration_manifest.json (optional)
+```
+
+`register_default_voice.py` only uses `reference_codes.npy` and does not load
+the optional encoder. To register a new voice from audio, use the service API
+after the optional `registration/` files have been downloaded.
+
+## CLI
+
+```bash
+bash run_infer.sh \
+ --text "这是一个中文测试" \
+ --voice default \
+ --max-new-tokens 128 \
+ --output outputs/test.wav
+```
+
+The command also writes `outputs/test.npy` with generated codes shaped
+`[10, frames]`.
+
+Set `ARKTTS_MODEL_DIR` or `ARKTTS_VOICES_DIR` to use another location. The
+runtime always selects `CPUExecutionProvider`; GPU execution is outside the
+scope of this export.
+
+## Local HTTP service
+
+```bash
+bash start_server.sh
+curl http://127.0.0.1:8024/api/health
+curl http://127.0.0.1:8024/api/tts \
+ -H 'Content-Type: application/json' \
+ -d '{"text":"你好,这是一个测试。","voice_name":"default"}' \
+ -o outputs/api.wav
+```
+
+The service exposes `/api/tts`, `/api/tts/stream`, `/api/tts/cancel`,
+`/api/voices`, `/api/voices/register`, `/api/registration/status`, and the
+OpenAI-compatible `/v1/audio/speech`. Stop it with `bash stop_server.sh`.
+
+## Verification
+
+```bash
+"$PWD/.venv/bin/python" -m pytest -q tests
+```
+
+`test_contract.py` checks the exact input/output names and shapes when
+`ARKTTS_MODEL_DIR` points at a downloaded model. Without model files it skips
+the ONNX-dependent check; the pure prompt and state-shape tests still run.
+
+The 0.1B model and this runtime are intended for local, authorized voice
+generation. Obtain consent before cloning a voice and disclose synthetic audio
+where appropriate.
diff --git a/onnx_runtime_0_1b_int8/README_zh.md b/onnx_runtime_0_1b_int8/README_zh.md
new file mode 100644
index 0000000..56d1f10
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/README_zh.md
@@ -0,0 +1,62 @@
+# Audio8 0.1B INT8 ONNX Runtime
+
+这是 `Audio8/audio8-TTS-0.1B-ONNX-INT8` 的独立 CPU ONNX Runtime。它放在
+单独目录中,因为仓库现有的 `onnx_runtime/` 是给 0.6B INT4 图使用的,不能
+直接处理 0.1B Falcon-H1 hybrid 图。
+
+0.1B Slow AR 每次只接受一个 `[1, 11, 1]` token column,并且要在调用之间保存:
+
+```text
+cache_keys [24, 1, 2, 2048, 64]
+cache_values [24, 1, 2, 2048, 64]
+conv_states [24, 1, 896, 4]
+ssm_states [24, 1, 24, 32, 64]
+```
+
+本实现会逐位置执行 prompt prefill,写回 Slow AR 返回的 state delta,处理
+4097 维 compact semantic logits,然后调用四层 Fast AR 和 FP16 codec decoder。
+
+## 安装
+
+支持 Python 3.10 及以上版本,Jetson issue 中的 Python 3.10 也可以使用:
+
+```bash
+python3 -m pip install -U "huggingface_hub[cli]"
+hf download Audio8/audio8-TTS-0.1B-ONNX-INT8 --local-dir model
+bash setup.sh
+python scripts/register_default_voice.py
+```
+
+模型文件放在本目录的 `model/` 下。`registration/` 下的 encoder 是可选的,
+只在从音频注册新音色时需要。默认音色脚本只读取 `reference_codes.npy`。
+
+## 命令行推理
+
+```bash
+bash run_infer.sh \
+ --text "这是一个中文测试" \
+ --voice default \
+ --max-new-tokens 128 \
+ --output outputs/test.wav
+```
+
+同时会生成 `[10, frames]` 形状的 `outputs/test.npy`。模型或音色放在其他位置时,
+可设置 `ARKTTS_MODEL_DIR`、`ARKTTS_VOICES_DIR`。
+
+## HTTP 服务
+
+```bash
+bash start_server.sh
+curl http://127.0.0.1:8024/api/health
+```
+
+服务提供 `/api/tts`、`/api/tts/stream`、`/api/tts/cancel`、音色查询和注册接口,
+以及兼容 OpenAI 的 `/v1/audio/speech`。使用 `bash stop_server.sh` 停止服务。
+
+## 测试
+
+```bash
+"$PWD/.venv/bin/python" -m pytest -q tests
+```
+
+请在获得授权后进行音色克隆,并在适当场景披露合成音频。
diff --git a/onnx_runtime_0_1b_int8/arktts_runtime/__init__.py b/onnx_runtime_0_1b_int8/arktts_runtime/__init__.py
new file mode 100644
index 0000000..622a289
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/arktts_runtime/__init__.py
@@ -0,0 +1,5 @@
+"""Standalone ONNX Runtime for Audio8 0.1B INT8 exports."""
+
+from .runtime import ArkTtsRuntime
+
+__all__ = ["ArkTtsRuntime"]
diff --git a/onnx_runtime_0_1b_int8/arktts_runtime/cli.py b/onnx_runtime_0_1b_int8/arktts_runtime/cli.py
new file mode 100644
index 0000000..bdf9843
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/arktts_runtime/cli.py
@@ -0,0 +1,55 @@
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+import numpy as np
+import soundfile as sf
+
+from .runtime import ArkTtsRuntime
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Audio8 0.1B INT8 ONNX Runtime")
+ parser.add_argument("--model-dir", type=Path, default=ROOT / "model")
+ parser.add_argument("--voices-dir", type=Path, default=ROOT / "voices")
+ parser.add_argument("--precision", choices=["int8"], default=None)
+ parser.add_argument("--codec-precision", choices=["fp16"], default=None)
+ parser.add_argument("--text", required=True)
+ parser.add_argument("--voice", default="default")
+ parser.add_argument("--output", type=Path, default=Path("output.wav"))
+ parser.add_argument("--max-new-tokens", type=int, default=256)
+ parser.add_argument("--temperature", type=float, default=0.7)
+ parser.add_argument("--top-p", type=float, default=0.9)
+ parser.add_argument("--top-k", type=int, default=50)
+ parser.add_argument("--seed", type=int, default=42)
+ parser.add_argument("--threads", type=int, default=5)
+ args = parser.parse_args()
+
+ runtime = ArkTtsRuntime(
+ args.model_dir,
+ args.voices_dir,
+ args.precision,
+ args.codec_precision,
+ args.threads,
+ )
+ audio, codes = runtime.synthesize(
+ text=args.text,
+ voice=args.voice,
+ max_new_tokens=args.max_new_tokens,
+ temperature=args.temperature,
+ top_p=args.top_p,
+ top_k=args.top_k,
+ seed=args.seed,
+ )
+ args.output.parent.mkdir(parents=True, exist_ok=True)
+ sf.write(str(args.output), audio, int(runtime.manifest["sample_rate"]))
+ np.save(args.output.with_suffix(".npy"), codes.astype(np.uint16))
+ print(f"saved {args.output}")
+ print(f"saved {args.output.with_suffix('.npy')}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/onnx_runtime_0_1b_int8/arktts_runtime/prompt.py b/onnx_runtime_0_1b_int8/arktts_runtime/prompt.py
new file mode 100644
index 0000000..37ca2a5
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/arktts_runtime/prompt.py
@@ -0,0 +1,93 @@
+from __future__ import annotations
+
+import re
+import unicodedata
+from pathlib import Path
+
+import numpy as np
+from tokenizers import Tokenizer
+
+_CJK_RANGES = (
+ "\u1100-\u11ff\u2e80-\u2fdf\u3000-\u303f\u3040-\u30ff\u3100-\u31ff"
+ "\u3400-\u4dbf\u4e00-\u9fff\ua960-\ua97f\uac00-\ud7a3\ud7b0-\ud7ff\uf900-\ufaff"
+ "\ufe30-\ufe4f\uff01-\uff9f\U00020000-\U0002fa1f"
+)
+_CJK_CHARACTER_RE = re.compile(rf"[{_CJK_RANGES}]")
+_LINE_BREAK_RE = re.compile(r"[\r\n\v\f\x1c-\x1e\x85\u2028\u2029]")
+
+
+def _normalize_whitespace(text: str) -> str:
+ def replace(match: re.Match[str]) -> str:
+ left = text[match.start() - 1] if match.start() else ""
+ right = text[match.end()] if match.end() < len(text) else ""
+ if (
+ _LINE_BREAK_RE.search(match.group())
+ and _CJK_CHARACTER_RE.fullmatch(left)
+ and _CJK_CHARACTER_RE.fullmatch(right)
+ ):
+ return ""
+ return " "
+
+ return re.sub(r"\s+", replace, text).strip()
+
+
+def clean_text(text: str) -> str:
+ value = "".join(
+ char
+ if char.isspace()
+ else ""
+ if unicodedata.category(char).startswith("C")
+ else char
+ for char in str(text)
+ )
+ return _normalize_whitespace(value)
+
+
+def format_reference_text(text: str) -> str:
+ text = clean_text(text)
+ return text if re.search(r"<\|speaker:\d+\|>", text) else f"<|speaker:0|>{text}"
+
+
+class PromptBuilder:
+ def __init__(self, tokenizer_dir: Path, semantic_begin_id: int, num_codebooks: int):
+ self.tokenizer = Tokenizer.from_file(str(Path(tokenizer_dir) / "tokenizer.json"))
+ self.semantic_begin_id = int(semantic_begin_id)
+ self.num_codebooks = int(num_codebooks)
+
+ def encode_text(self, text: str) -> list[int]:
+ return list(self.tokenizer.encode(text, add_special_tokens=False).ids)
+
+ def build(self, target_text: str, reference_text: str, reference_codes: np.ndarray) -> np.ndarray:
+ target = clean_text(target_text)
+ if not target:
+ raise ValueError("text must not be empty")
+ codes = np.asarray(reference_codes, dtype=np.int64)
+ if codes.ndim != 2 or codes.shape[0] != self.num_codebooks or codes.shape[1] == 0:
+ raise ValueError(
+ f"reference codes must have shape [{self.num_codebooks}, T>0], got {codes.shape}"
+ )
+ if int(codes.min()) < 0:
+ raise ValueError("reference codes must be non-negative")
+
+ prefix_parts = [
+ "<|im_start|>system\n",
+ "convert the provided text to speech reference to the following:\n\nText:\n",
+ format_reference_text(reference_text),
+ "\n\nSpeech:\n",
+ ]
+ suffix_parts = [
+ "<|im_end|>\n",
+ "<|im_start|>user\n",
+ target,
+ "<|im_end|>\n",
+ "<|im_start|>assistant\n<|voice|>",
+ ]
+ prefix = [token for part in prefix_parts for token in self.encode_text(part)]
+ suffix = [token for part in suffix_parts for token in self.encode_text(part)]
+ semantic_ids = (codes[0] + self.semantic_begin_id).tolist()
+ row0 = np.asarray(prefix + semantic_ids + suffix, dtype=np.int64)
+ values = np.zeros((self.num_codebooks + 1, row0.size), dtype=np.int64)
+ values[0] = row0
+ reference_start = len(prefix)
+ values[1:, reference_start : reference_start + codes.shape[1]] = codes
+ return values[np.newaxis]
diff --git a/onnx_runtime_0_1b_int8/arktts_runtime/registration.py b/onnx_runtime_0_1b_int8/arktts_runtime/registration.py
new file mode 100644
index 0000000..10c5f23
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/arktts_runtime/registration.py
@@ -0,0 +1,162 @@
+from __future__ import annotations
+
+import gc
+import hashlib
+import io
+import json
+import os
+import shutil
+import tempfile
+from datetime import datetime, timezone
+from math import gcd
+from pathlib import Path
+
+import numpy as np
+import onnxruntime as ort
+import soundfile as sf
+
+MAX_AUDIO_BYTES = 50 * 1024 * 1024
+MAX_AUDIO_SECONDS = 30.0
+MIN_AUDIO_SECONDS = 0.5
+
+
+class VoiceRegistration:
+ """Optional reference-audio encoder used to create a voice profile."""
+
+ def __init__(self, registration_dir: Path, voices_root: Path, model_fingerprint: str):
+ self.root = Path(registration_dir).resolve()
+ self.voices_root = Path(voices_root).resolve()
+ self.encoder_path = self.root / "codec_encoder_fp16.onnx"
+ self.manifest_path = self.root / "registration_manifest.json"
+ self.expected_fingerprint = model_fingerprint
+
+ def status(self) -> dict[str, str | bool | None]:
+ available = self.encoder_path.is_file() and self.manifest_path.is_file()
+ reason: str | None = None
+ if available:
+ try:
+ manifest = json.loads(self.manifest_path.read_text(encoding="utf-8"))
+ if manifest.get("model_fingerprint") != self.expected_fingerprint:
+ available = False
+ reason = "registration model fingerprint does not match online model"
+ except (OSError, json.JSONDecodeError):
+ available = False
+ reason = "registration manifest is invalid"
+ else:
+ reason = "registration package was not found"
+ return {"available": available, "reason": reason}
+
+ @staticmethod
+ def _validate_name(name: str) -> str:
+ value = name.strip()
+ if not value or len(value) > 64 or value in {".", ".."} or Path(value).name != value:
+ raise ValueError("voice name must be one path component with at most 64 characters")
+ return value
+
+ @staticmethod
+ def _decode_audio(data: bytes, target_rate: int) -> tuple[np.ndarray, int]:
+ if not data or len(data) > MAX_AUDIO_BYTES:
+ raise ValueError("audio file must be between 1 byte and 50 MiB")
+ try:
+ audio, sample_rate = sf.read(io.BytesIO(data), dtype="float32", always_2d=True)
+ except (RuntimeError, sf.LibsndfileError) as exc:
+ raise ValueError(f"unsupported or invalid audio file: {exc}") from exc
+ audio = audio.mean(axis=1)
+ source_rate = int(sample_rate)
+ duration = audio.size / max(1, source_rate)
+ if duration < MIN_AUDIO_SECONDS or duration > MAX_AUDIO_SECONDS:
+ raise ValueError("reference audio duration must be between 0.5 and 30 seconds")
+ if not np.isfinite(audio).all():
+ raise ValueError("reference audio contains non-finite samples")
+ if source_rate != int(target_rate):
+ from scipy.signal import resample_poly
+
+ factor = gcd(source_rate, int(target_rate))
+ audio = resample_poly(
+ audio,
+ int(target_rate) // factor,
+ source_rate // factor,
+ ).astype(np.float32)
+ padding = (-audio.size) % 2048
+ if padding:
+ audio = np.pad(audio, (0, padding))
+ return np.ascontiguousarray(audio.reshape(1, 1, -1)), source_rate
+
+ def register(
+ self,
+ data: bytes,
+ filename: str,
+ text: str,
+ name: str,
+ overwrite: bool = False,
+ ) -> dict:
+ state = self.status()
+ if not state["available"]:
+ raise RuntimeError(str(state["reason"]))
+ voice_name = self._validate_name(name)
+ reference_text = " ".join(text.strip().split())
+ if not reference_text:
+ raise ValueError("reference text must not be empty")
+ manifest = json.loads(self.manifest_path.read_text(encoding="utf-8"))
+ audio, source_rate = self._decode_audio(data, int(manifest["sample_rate"]))
+ target = self.voices_root / voice_name
+ if target.exists() and not overwrite:
+ raise FileExistsError(f"voice already exists: {voice_name}")
+
+ options = ort.SessionOptions()
+ options.log_severity_level = 3
+ session = None
+ try:
+ session = ort.InferenceSession(
+ str(self.encoder_path),
+ sess_options=options,
+ providers=["CPUExecutionProvider"],
+ )
+ input_info = session.get_inputs()[0]
+ input_type = np.float16 if input_info.type == "tensor(float16)" else np.float32
+ codes = np.asarray(session.run(None, {input_info.name: audio.astype(input_type)})[0])
+ finally:
+ del session
+ gc.collect()
+
+ if codes.ndim == 3:
+ codes = codes[0]
+ expected_codebooks = int(manifest["num_codebooks"])
+ codebook_size = int(manifest.get("codebook_size", 4096))
+ if (
+ codes.ndim != 2
+ or codes.shape[0] != expected_codebooks
+ or codes.shape[1] == 0
+ or int(codes.min()) < 0
+ or int(codes.max()) >= codebook_size
+ ):
+ raise RuntimeError(f"encoder returned invalid codes: {codes.shape}")
+
+ self.voices_root.mkdir(parents=True, exist_ok=True)
+ temp = Path(tempfile.mkdtemp(prefix=f".{voice_name}.", dir=self.voices_root))
+ try:
+ np.save(temp / "codes.npy", codes.astype(np.uint16))
+ meta = {
+ "name": voice_name,
+ "reference_text": reference_text,
+ "shape": list(codes.shape),
+ "dtype": "uint16",
+ "sample_rate": int(manifest["sample_rate"]),
+ "source_audio": Path(filename or "reference_audio").name,
+ "source_sample_rate": source_rate,
+ "source_sha256": hashlib.sha256(data).hexdigest(),
+ "model_fingerprint": manifest["model_fingerprint"],
+ "created_at": datetime.now(timezone.utc).isoformat(),
+ "source_kind": "onnx_runtime_registration",
+ }
+ (temp / "meta.json").write_text(
+ json.dumps(meta, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ if target.exists():
+ shutil.rmtree(target)
+ os.replace(temp, target)
+ finally:
+ if temp.exists():
+ shutil.rmtree(temp)
+ return meta
diff --git a/onnx_runtime_0_1b_int8/arktts_runtime/runtime.py b/onnx_runtime_0_1b_int8/arktts_runtime/runtime.py
new file mode 100644
index 0000000..c9b2952
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/arktts_runtime/runtime.py
@@ -0,0 +1,481 @@
+from __future__ import annotations
+
+import json
+import threading
+from collections.abc import Iterator
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+import onnxruntime as ort
+
+from .prompt import PromptBuilder
+from .voices import VoiceStore
+
+
+ORT_DTYPES: dict[str, Any] = {
+ "tensor(float)": np.float32,
+ "tensor(float16)": np.float16,
+ "tensor(int64)": np.int64,
+ "tensor(int32)": np.int32,
+ "tensor(bool)": np.bool_,
+}
+
+
+def _session(path: Path, threads: int | None = None) -> ort.InferenceSession:
+ if not path.is_file():
+ raise FileNotFoundError(f"ONNX model was not found: {path}")
+ options = ort.SessionOptions()
+ options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
+ options.log_severity_level = 3
+ if threads is not None:
+ options.intra_op_num_threads = max(1, int(threads))
+ options.inter_op_num_threads = max(1, int(threads) // 2)
+ return ort.InferenceSession(
+ str(path),
+ sess_options=options,
+ providers=["CPUExecutionProvider"],
+ )
+
+
+def _sample(
+ logits: np.ndarray,
+ temperature: float,
+ top_p: float,
+ top_k: int,
+ rng: np.random.Generator,
+) -> int:
+ values = np.asarray(logits, dtype=np.float64).reshape(-1)
+ if values.size == 0 or not np.isfinite(values).any():
+ raise ValueError("model returned empty or non-finite logits")
+ temperature = max(float(temperature), 1e-5)
+ top_p = min(max(float(top_p), 1e-5), 1.0)
+ top_k = min(max(int(top_k), 1), values.size)
+
+ order = np.argsort(values)[::-1]
+ sorted_values = values[order]
+ probabilities = np.exp(sorted_values - np.max(sorted_values))
+ probabilities /= probabilities.sum()
+ cumulative = np.cumsum(probabilities)
+ remove = (cumulative > top_p) | (np.arange(values.size) >= top_k)
+ remove[0] = False
+ filtered = values.copy()
+ filtered[order[remove]] = -np.inf
+ scaled = filtered / temperature
+ scaled -= np.max(scaled)
+ probabilities = np.exp(scaled)
+ probabilities /= probabilities.sum()
+ # Gumbel-max is stable and makes the RNG sequence explicit for reproducible CLI runs.
+ uniform = np.clip(rng.random(probabilities.size), 1e-12, 1.0 - 1e-12)
+ gumbel = -np.log(-np.log(uniform))
+ return int(np.argmax(np.log(np.clip(probabilities, 1e-300, 1.0)) + gumbel))
+
+
+def _dtype(value: str) -> np.dtype:
+ try:
+ return np.dtype(ORT_DTYPES[value])
+ except KeyError as exc:
+ raise ValueError(f"unsupported ONNX tensor type: {value}") from exc
+
+
+def _shape(info: ort.NodeArg, fallback: tuple[int, ...]) -> tuple[int, ...]:
+ """Resolve symbolic dimensions to the dimensions used by this export."""
+ result: list[int] = []
+ for index, dimension in enumerate(info.shape):
+ if isinstance(dimension, int) and dimension > 0:
+ result.append(dimension)
+ elif index < len(fallback):
+ result.append(int(fallback[index]))
+ else:
+ raise ValueError(f"cannot resolve dynamic shape for {info.name}: {info.shape}")
+ return tuple(result)
+
+
+class ArkTtsRuntime:
+ """CPU runtime for ``Audio8/audio8-TTS-0.1B-ONNX-INT8``.
+
+ The 0.1B graph is a Falcon-H1 hybrid export. Its slow graph consumes one
+ ``[1, 11, 1]`` column at a time and returns state deltas; it is not
+ compatible with the 0.6B runtime's independent per-layer KV inputs.
+ """
+
+ _SLOW_INPUTS = {
+ "codes",
+ "position",
+ "cache_keys",
+ "cache_values",
+ "conv_states",
+ "ssm_states",
+ }
+ _FAST_INPUTS = {
+ "slow_hidden",
+ "token_id",
+ "use_slow_hidden",
+ "input_pos",
+ "cache_key_0",
+ "cache_key_1",
+ "cache_key_2",
+ "cache_key_3",
+ "cache_value_0",
+ "cache_value_1",
+ "cache_value_2",
+ "cache_value_3",
+ }
+
+ def __init__(
+ self,
+ model_dir: Path,
+ voices_dir: Path,
+ precision: str | None = None,
+ codec_precision: str | None = None,
+ threads: int | None = None,
+ ):
+ self.model_dir = Path(model_dir).resolve()
+ manifest_path = self.model_dir / "runtime_manifest.json"
+ if not manifest_path.is_file():
+ raise FileNotFoundError(f"runtime manifest was not found: {manifest_path}")
+ self.manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+ self.precision = precision or self.manifest["default_precision"]
+ if self.precision not in self.manifest["available_precisions"]:
+ raise ValueError(f"unsupported precision: {self.precision}")
+ self.codec_precision = codec_precision or self.manifest.get(
+ "default_codec_precision", "fp16"
+ )
+ available_codec = self.manifest.get("available_codec_precisions", ["fp16"])
+ if self.codec_precision not in available_codec:
+ raise ValueError(f"unsupported codec precision: {self.codec_precision}")
+
+ slow_name = self.manifest.get("slow_decode_models", {}).get(
+ self.precision,
+ self.manifest.get("slow_decode_model", f"slow_ar_{self.precision}.onnx"),
+ )
+ fast_name = self.manifest.get("fast_models", {}).get(
+ self.precision,
+ self.manifest.get("fast_model", f"fast_ar_{self.precision}.onnx"),
+ )
+ codec_name = self.manifest.get("codec_models", {}).get(
+ self.codec_precision,
+ f"codec_decoder_{self.codec_precision}.onnx",
+ )
+ self.slow = _session(self.model_dir / slow_name, threads)
+ self.fast = _session(self.model_dir / fast_name, threads)
+ self.decoder = _session(self.model_dir / codec_name, threads)
+
+ self.slow_inputs = {item.name: item for item in self.slow.get_inputs()}
+ self.fast_inputs = {item.name: item for item in self.fast.get_inputs()}
+ self._validate_contract()
+ self._slow_outputs = {item.name: index for index, item in enumerate(self.slow.get_outputs())}
+ self._fast_outputs = {item.name: index for index, item in enumerate(self.fast.get_outputs())}
+
+ self.prompt_builder = PromptBuilder(
+ self.model_dir / "tokenizer",
+ int(self.manifest["semantic_begin_id"]),
+ int(self.manifest["num_codebooks"]),
+ )
+ self.voices = VoiceStore(Path(voices_dir), int(self.manifest["num_codebooks"]))
+
+ def _validate_contract(self) -> None:
+ slow_names = set(self.slow_inputs)
+ missing = self._SLOW_INPUTS - slow_names
+ if missing:
+ raise ValueError(
+ "this runtime expects the 0.1B hybrid slow graph; "
+ f"missing inputs: {sorted(missing)}"
+ )
+ fast_missing = self._FAST_INPUTS - set(self.fast_inputs)
+ if fast_missing:
+ raise ValueError(f"0.1B fast graph is missing inputs: {sorted(fast_missing)}")
+ if tuple(self.slow_inputs["codes"].shape[-2:]) != (11, 1):
+ raise ValueError(f"unexpected slow codes shape: {self.slow_inputs['codes'].shape}")
+ if tuple(self.slow_inputs["position"].shape) not in ((1,), (None,)):
+ raise ValueError(f"unexpected slow position shape: {self.slow_inputs['position'].shape}")
+ if int(self.manifest["num_layers"]) != int(self.slow_inputs["cache_keys"].shape[0]):
+ raise ValueError("runtime_manifest num_layers does not match slow cache_keys")
+ if int(self.manifest["num_fast_layers"]) != 4:
+ raise ValueError("the 0.1B fast export must have four cache layers")
+
+ def _empty_slow_state(self) -> dict[str, np.ndarray]:
+ specs = {
+ "cache_keys": (
+ int(self.manifest["num_layers"]),
+ 1,
+ int(self.manifest["n_local_heads"]),
+ int(self.manifest["max_seq_len"]),
+ int(self.manifest["head_dim"]),
+ ),
+ "cache_values": (
+ int(self.manifest["num_layers"]),
+ 1,
+ int(self.manifest["n_local_heads"]),
+ int(self.manifest["max_seq_len"]),
+ int(self.manifest["head_dim"]),
+ ),
+ "conv_states": (
+ int(self.manifest["num_layers"]),
+ 1,
+ 896,
+ int(self.manifest["mamba_d_conv"]),
+ ),
+ "ssm_states": (
+ int(self.manifest["num_layers"]),
+ 1,
+ int(self.manifest["mamba_n_heads"]),
+ int(self.manifest["mamba_d_head"]),
+ int(self.manifest["mamba_d_state"]),
+ ),
+ }
+ return {
+ name: np.zeros(
+ _shape(self.slow_inputs[name], fallback),
+ dtype=_dtype(self.slow_inputs[name].type),
+ )
+ for name, fallback in specs.items()
+ }
+
+ def _empty_fast_caches(self) -> dict[str, np.ndarray]:
+ result: dict[str, np.ndarray] = {}
+ for name, info in self.fast_inputs.items():
+ if not (name.startswith("cache_key_") or name.startswith("cache_value_")):
+ continue
+ fallback = (1, 2, int(self.manifest["num_codebooks"]), int(self.manifest["fast_head_dim"]))
+ result[name] = np.zeros(_shape(info, fallback), dtype=_dtype(info.type))
+ return result
+
+ @staticmethod
+ def _output(outputs: list[np.ndarray], names: dict[str, int], name: str, index: int) -> np.ndarray:
+ output_index = names.get(name, index)
+ if output_index >= len(outputs):
+ raise ValueError(f"ONNX graph did not return expected output {name}")
+ return np.asarray(outputs[output_index])
+
+ def _slow_step(
+ self,
+ codes: np.ndarray,
+ position: int,
+ state: dict[str, np.ndarray],
+ ) -> tuple[np.ndarray, np.ndarray]:
+ codes = np.asarray(codes, dtype=np.int64)
+ if codes.shape != (1, int(self.manifest["num_codebooks"]) + 1, 1):
+ raise ValueError(f"slow codes must have shape [1, 11, 1], got {codes.shape}")
+ position_value = np.asarray([int(position)], dtype=np.int64)
+ feeds = {
+ "codes": codes,
+ "position": position_value,
+ "cache_keys": state["cache_keys"],
+ "cache_values": state["cache_values"],
+ "conv_states": state["conv_states"],
+ "ssm_states": state["ssm_states"],
+ }
+ outputs = self.slow.run(None, feeds)
+ key_delta = self._output(outputs, self._slow_outputs, "key_delta", 2)
+ value_delta = self._output(outputs, self._slow_outputs, "value_delta", 3)
+ next_conv = self._output(outputs, self._slow_outputs, "next_conv_states", 4)
+ next_ssm = self._output(outputs, self._slow_outputs, "next_ssm_states", 5)
+ if not 0 <= int(position) < state["cache_keys"].shape[3]:
+ raise ValueError(f"slow position {position} exceeds cache length")
+ state["cache_keys"][:, :, :, int(position), :] = key_delta
+ state["cache_values"][:, :, :, int(position), :] = value_delta
+ state["conv_states"] = next_conv.astype(state["conv_states"].dtype, copy=False)
+ state["ssm_states"] = next_ssm.astype(state["ssm_states"].dtype, copy=False)
+ logits = self._output(outputs, self._slow_outputs, "logits", 0)
+ hidden = self._output(outputs, self._slow_outputs, "hidden", 1)
+ return logits[0, -1], hidden[:, -1:, :]
+
+ def _fast_step(
+ self,
+ hidden: np.ndarray,
+ token_id: int,
+ use_slow_hidden: bool,
+ position: int,
+ caches: dict[str, np.ndarray],
+ ) -> np.ndarray:
+ hidden_info = self.fast_inputs["slow_hidden"]
+ use_info = self.fast_inputs["use_slow_hidden"]
+ feeds: dict[str, np.ndarray] = {
+ "slow_hidden": np.asarray(hidden, dtype=_dtype(hidden_info.type)),
+ "token_id": np.asarray([[int(token_id)]], dtype=_dtype(self.fast_inputs["token_id"].type)),
+ "use_slow_hidden": np.asarray(
+ [bool(use_slow_hidden)], dtype=_dtype(use_info.type)
+ ),
+ "input_pos": np.asarray([int(position)], dtype=_dtype(self.fast_inputs["input_pos"].type)),
+ }
+ feeds.update(caches)
+ outputs = self.fast.run(None, feeds)
+ for index in range(4):
+ key_name = f"cache_key_{index}"
+ value_name = f"cache_value_{index}"
+ key_delta = self._output(
+ outputs,
+ self._fast_outputs,
+ f"key_delta_{index}",
+ 1 + index * 2,
+ )
+ value_delta = self._output(
+ outputs,
+ self._fast_outputs,
+ f"value_delta_{index}",
+ 2 + index * 2,
+ )
+ self._update_fast_cache(caches[key_name], key_delta, position)
+ self._update_fast_cache(caches[value_name], value_delta, position)
+ logits = self._output(outputs, self._fast_outputs, "logits", 0)
+ return logits[0, -1]
+
+ @staticmethod
+ def _update_fast_cache(cache: np.ndarray, value: np.ndarray, position: int) -> None:
+ value = np.asarray(value, dtype=cache.dtype)
+ if value.shape == cache.shape:
+ cache[...] = value
+ return
+ if value.ndim == cache.ndim and value.shape[:2] == cache.shape[:2] and value.shape[3:] == cache.shape[3:]:
+ length = value.shape[2]
+ cache[:, :, int(position) : int(position) + length, :] = value
+ return
+ raise ValueError(f"unexpected fast cache output shape {value.shape}; expected {cache.shape}")
+
+ def _sample_semantic(
+ self,
+ logits: np.ndarray,
+ previous: list[int],
+ temperature: float,
+ top_p: float,
+ top_k: int,
+ rng: np.random.Generator,
+ ) -> int:
+ begin = int(self.manifest["semantic_begin_id"])
+ end = int(self.manifest["semantic_end_id"])
+ stop = int(self.manifest["im_end_id"])
+ expected = int(self.manifest["codebook_size"]) + 1
+ values = np.asarray(logits, dtype=np.float32).reshape(-1)
+ if self.manifest.get("slow_logits_layout") != "relative_semantic_then_eos":
+ raise ValueError("this runtime only supports relative_semantic_then_eos logits")
+ if values.size != expected:
+ raise ValueError(f"unexpected slow logits size: {values.size}, expected {expected}")
+ normal_index = _sample(values, temperature, top_p, top_k, rng)
+ high_index = _sample(values, 1.0, 0.9, top_k, rng)
+ normal = stop if normal_index == expected - 1 else begin + normal_index
+ high = stop if high_index == expected - 1 else begin + high_index
+ if begin <= normal <= end and normal in previous:
+ return high
+ return normal
+
+ def iter_codes(
+ self,
+ text: str,
+ voice: str,
+ max_new_tokens: int = 1024,
+ temperature: float = 0.7,
+ top_p: float = 0.9,
+ top_k: int = 50,
+ seed: int = 42,
+ stop_event: threading.Event | None = None,
+ ) -> Iterator[np.ndarray]:
+ reference_codes, meta = self.voices.load(voice)
+ prompt = self.prompt_builder.build(text, meta["reference_text"], reference_codes)
+ prompt_len = int(prompt.shape[2])
+ max_seq_len = int(self.manifest["max_seq_len"])
+ if prompt_len >= max_seq_len:
+ raise ValueError(f"prompt length {prompt_len} exceeds max sequence length {max_seq_len}")
+ max_new_tokens = min(max(0, int(max_new_tokens)), max_seq_len - prompt_len)
+
+ rng = np.random.default_rng(int(seed))
+ slow_state = self._empty_slow_state()
+ logits: np.ndarray | None = None
+ hidden: np.ndarray | None = None
+ # The exported hybrid graph is a one-token graph, including prefill.
+ for position in range(prompt_len):
+ logits, hidden = self._slow_step(prompt[:, :, position : position + 1], position, slow_state)
+
+ previous: list[int] = []
+ begin = int(self.manifest["semantic_begin_id"])
+ stop = int(self.manifest["im_end_id"])
+ codebook_size = int(self.manifest["codebook_size"])
+ assert logits is not None and hidden is not None
+ for step in range(max_new_tokens):
+ if stop_event is not None and stop_event.is_set():
+ return
+ semantic = self._sample_semantic(logits, previous, temperature, top_p, top_k, rng)
+ if semantic == stop:
+ return
+ previous.append(semantic)
+ previous = previous[-10:]
+
+ fast_caches = self._empty_fast_caches()
+ self._fast_step(hidden, 0, True, 0, fast_caches)
+ first_code = semantic - begin
+ if not 0 <= first_code < codebook_size:
+ raise ValueError(f"semantic token {semantic} is outside the codebook range")
+ codebooks = [first_code]
+ token = first_code
+ for fast_position in range(1, int(self.manifest["num_codebooks"])):
+ fast_logits = self._fast_step(hidden, token, False, fast_position, fast_caches)
+ token = _sample(fast_logits, temperature, top_p, top_k, rng)
+ codebooks.append(token)
+ frame = np.asarray(codebooks, dtype=np.int64)
+ yield frame
+ if step + 1 >= max_new_tokens:
+ return
+ column = np.concatenate([[semantic], frame]).reshape(1, -1, 1)
+ logits, hidden = self._slow_step(column, prompt_len + step, slow_state)
+
+ def decode_codes(self, codes: np.ndarray) -> np.ndarray:
+ values = np.asarray(codes, dtype=np.int64)
+ if values.ndim == 2:
+ values = values[np.newaxis]
+ expected = int(self.manifest["num_codebooks"])
+ if values.ndim != 3 or values.shape[1] != expected or values.shape[2] == 0:
+ raise ValueError(f"invalid generated codes shape: {values.shape}")
+ decoder_input = self.decoder.get_inputs()[0]
+ values = values.astype(_dtype(decoder_input.type), copy=False)
+ audio = self.decoder.run(None, {decoder_input.name: values})[0]
+ return np.asarray(audio, dtype=np.float32).reshape(-1)
+
+ def synthesize(self, **kwargs: Any) -> tuple[np.ndarray, np.ndarray]:
+ frames = list(self.iter_codes(**kwargs))
+ if not frames:
+ raise RuntimeError("model produced no codec frames")
+ codes = np.stack(frames, axis=1)
+ return self.decode_codes(codes), codes
+
+ def stream(self, chunk_frames: int = 12, **kwargs: Any):
+ all_frames: list[np.ndarray] = []
+ emitted_samples = 0
+ hop = int(self.manifest["codec_hop_length"])
+ context = int(self.manifest.get("stream_context_frames", 128))
+ guard = int(self.manifest.get("stream_guard_frames", 1)) * hop
+ sequence = 0
+ for frame in self.iter_codes(**kwargs):
+ all_frames.append(frame)
+ if len(all_frames) % max(1, int(chunk_frames)) != 0:
+ continue
+ start_frame = max(0, len(all_frames) - context - int(chunk_frames))
+ window = np.stack(all_frames[start_frame:], axis=1)
+ audio = self.decode_codes(window)
+ absolute_start = start_frame * hop
+ stable_end = absolute_start + max(0, audio.size - guard)
+ begin = max(0, emitted_samples - absolute_start)
+ end = max(begin, stable_end - absolute_start)
+ if end > begin:
+ chunk = np.ascontiguousarray(audio[begin:end])
+ emitted_samples += chunk.size
+ yield {
+ "type": "audio_chunk",
+ "seq": sequence,
+ "audio": chunk,
+ "frame_count": len(all_frames),
+ }
+ sequence += 1
+ if not all_frames:
+ raise RuntimeError("model produced no codec frames")
+ start_frame = max(0, len(all_frames) - context - int(chunk_frames))
+ audio = self.decode_codes(np.stack(all_frames[start_frame:], axis=1))
+ absolute_start = start_frame * hop
+ begin = max(0, emitted_samples - absolute_start)
+ if begin < audio.size:
+ yield {
+ "type": "audio_chunk",
+ "seq": sequence,
+ "audio": np.ascontiguousarray(audio[begin:]),
+ "frame_count": len(all_frames),
+ }
+ yield {"type": "complete", "codes": np.stack(all_frames, axis=1)}
diff --git a/onnx_runtime_0_1b_int8/arktts_runtime/service.py b/onnx_runtime_0_1b_int8/arktts_runtime/service.py
new file mode 100644
index 0000000..21a095f
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/arktts_runtime/service.py
@@ -0,0 +1,245 @@
+from __future__ import annotations
+
+import base64
+import gc
+import io
+import json
+import os
+import threading
+from pathlib import Path
+
+import numpy as np
+import soundfile as sf
+from fastapi import FastAPI, File, Form, HTTPException, UploadFile
+from fastapi.responses import Response, StreamingResponse
+from pydantic import BaseModel, Field
+
+from .registration import MAX_AUDIO_BYTES, VoiceRegistration
+from .runtime import ArkTtsRuntime
+
+ROOT = Path(__file__).resolve().parents[1]
+MODEL_DIR = Path(os.getenv("ARKTTS_MODEL_DIR", str(ROOT / "model"))).expanduser()
+VOICES_DIR = Path(os.getenv("ARKTTS_VOICES_DIR", str(ROOT / "voices"))).expanduser()
+REGISTRATION_DIR = Path(
+ os.getenv("ARKTTS_REGISTRATION_DIR", str(MODEL_DIR / "registration"))
+).expanduser()
+PRECISION = os.getenv("ARKTTS_PRECISION") or None
+CODEC_PRECISION = os.getenv("ARKTTS_CODEC_PRECISION") or None
+THREADS = int(os.getenv("ARKTTS_THREADS", "5"))
+
+app = FastAPI(title="Audio8 0.1B INT8 ONNX Runtime")
+runtime: ArkTtsRuntime | None = None
+registration: VoiceRegistration | None = None
+request_lock = threading.RLock()
+active_stop: threading.Event | None = None
+
+
+class TtsRequest(BaseModel):
+ text: str = Field(..., min_length=1, max_length=1000)
+ voice_name: str = "default"
+ max_new_tokens: int = Field(256, ge=1, le=2048)
+ temperature: float = Field(0.7, gt=0.0, le=2.0)
+ top_p: float = Field(0.9, gt=0.0, le=1.0)
+ top_k: int = Field(50, ge=1, le=4096)
+ seed: int = 42
+
+
+class OpenAiSpeechRequest(BaseModel):
+ model: str = "arktts-0.1b"
+ input: str = Field(..., min_length=1, max_length=1000)
+ voice: str = "default"
+ response_format: str = "wav"
+
+
+def _load_runtime() -> ArkTtsRuntime:
+ global runtime, registration
+ runtime = ArkTtsRuntime(MODEL_DIR, VOICES_DIR, PRECISION, CODEC_PRECISION, THREADS)
+ registration = VoiceRegistration(
+ REGISTRATION_DIR,
+ runtime.voices.root,
+ str(runtime.manifest["model_fingerprint"]),
+ )
+ return runtime
+
+
[email protected]_event("startup")
+def startup() -> None:
+ _load_runtime()
+
+
+def require_runtime() -> ArkTtsRuntime:
+ if runtime is None:
+ raise HTTPException(503, "runtime is not loaded")
+ return runtime
+
+
+def _wav_response(audio: np.ndarray, sample_rate: int) -> Response:
+ buffer = io.BytesIO()
+ sf.write(buffer, np.asarray(audio, dtype=np.float32), int(sample_rate), format="WAV")
+ return Response(buffer.getvalue(), media_type="audio/wav")
+
+
[email protected]("/api/health")
+def health() -> dict:
+ obj = require_runtime()
+ return {
+ "ok": True,
+ "model": obj.manifest.get("model_id"),
+ "precision": obj.precision,
+ "codec_precision": obj.codec_precision,
+ "providers": {
+ "slow": obj.slow.get_providers(),
+ "fast": obj.fast.get_providers(),
+ "decoder": obj.decoder.get_providers(),
+ },
+ }
+
+
[email protected]("/api/voices")
+def voices() -> dict:
+ return {"voices": require_runtime().voices.list()}
+
+
[email protected]("/api/registration/status")
+def registration_status() -> dict:
+ if registration is None:
+ raise HTTPException(503, "registration is not initialized")
+ return registration.status()
+
+
[email protected]("/api/voices/register")
+def register_voice(
+ audio: UploadFile = File(...),
+ text: str = Form(...),
+ name: str = Form(...),
+ overwrite: bool = Form(False),
+) -> dict:
+ global runtime
+ if registration is None:
+ raise HTTPException(503, "registration is not initialized")
+ data = audio.file.read(MAX_AUDIO_BYTES + 1)
+ with request_lock:
+ current = require_runtime()
+ precision, codec_precision = current.precision, current.codec_precision
+ runtime = None
+ del current
+ gc.collect()
+ try:
+ meta = registration.register(
+ data,
+ audio.filename or "reference_audio",
+ text,
+ name,
+ overwrite,
+ )
+ except FileExistsError as exc:
+ raise HTTPException(409, str(exc)) from exc
+ except ValueError as exc:
+ raise HTTPException(400, str(exc)) from exc
+ except RuntimeError as exc:
+ raise HTTPException(503, str(exc)) from exc
+ finally:
+ runtime = ArkTtsRuntime(MODEL_DIR, VOICES_DIR, precision, codec_precision, THREADS)
+ gc.collect()
+ return {"ok": True, "voice": meta}
+
+
[email protected]("/api/tts")
+def tts(request: TtsRequest) -> Response:
+ with request_lock:
+ obj = require_runtime()
+ audio, _ = obj.synthesize(
+ text=request.text,
+ voice=request.voice_name,
+ max_new_tokens=request.max_new_tokens,
+ temperature=request.temperature,
+ top_p=request.top_p,
+ top_k=request.top_k,
+ seed=request.seed,
+ )
+ return _wav_response(audio, int(obj.manifest["sample_rate"]))
+
+
[email protected]("/v1/audio/speech")
+def openai_speech(request: OpenAiSpeechRequest) -> Response:
+ if request.model not in {"arktts", "arktts-0.1b", "tts-1"}:
+ raise HTTPException(400, f"unsupported model: {request.model}")
+ if request.response_format not in {"wav", "pcm"}:
+ raise HTTPException(400, "response_format must be wav or pcm")
+ with request_lock:
+ obj = require_runtime()
+ audio, _ = obj.synthesize(text=request.input, voice=request.voice)
+ if request.response_format == "pcm":
+ pcm = (np.clip(audio, -1.0, 1.0) * 32767.0).astype("<i2").tobytes()
+ return Response(pcm, media_type="application/octet-stream")
+ return _wav_response(audio, int(obj.manifest["sample_rate"]))
+
+
[email protected]("/api/tts/stream")
+def stream_tts(request: TtsRequest) -> StreamingResponse:
+ def events():
+ global active_stop
+ stop = threading.Event()
+ active_stop = stop
+ try:
+ with request_lock:
+ obj = require_runtime()
+ yield json.dumps(
+ {
+ "event": "start",
+ "sample_rate": int(obj.manifest["sample_rate"]),
+ "channels": 1,
+ "sample_format": "s16le",
+ }
+ ) + "\n"
+ for event in obj.stream(
+ text=request.text,
+ voice=request.voice_name,
+ max_new_tokens=request.max_new_tokens,
+ temperature=request.temperature,
+ top_p=request.top_p,
+ top_k=request.top_k,
+ seed=request.seed,
+ stop_event=stop,
+ ):
+ if stop.is_set():
+ yield json.dumps({"event": "cancelled"}) + "\n"
+ return
+ if event["type"] == "audio_chunk":
+ pcm = (np.clip(event["audio"], -1.0, 1.0) * 32767.0).astype("<i2")
+ yield json.dumps(
+ {
+ "event": "audio_chunk",
+ "seq": int(event["seq"]),
+ "frame_count": int(event["frame_count"]),
+ "pcm_b64": base64.b64encode(pcm.tobytes()).decode("ascii"),
+ }
+ ) + "\n"
+ else:
+ yield json.dumps(
+ {"event": "complete", "frame_count": int(event["codes"].shape[1])}
+ ) + "\n"
+ finally:
+ if active_stop is stop:
+ active_stop = None
+
+ return StreamingResponse(events(), media_type="application/x-ndjson")
+
+
[email protected]("/api/tts/cancel")
+def cancel() -> dict[str, bool]:
+ if active_stop is None:
+ return {"ok": True, "cancelled": False}
+ active_stop.set()
+ return {"ok": True, "cancelled": True}
+
+
[email protected]("/api/runtime/reload")
+def reload_runtime() -> dict:
+ global runtime, registration
+ with request_lock:
+ runtime = None
+ registration = None
+ gc.collect()
+ obj = _load_runtime()
+ return {"ok": True, "precision": obj.precision, "codec_precision": obj.codec_precision}
diff --git a/onnx_runtime_0_1b_int8/arktts_runtime/voices.py b/onnx_runtime_0_1b_int8/arktts_runtime/voices.py
new file mode 100644
index 0000000..b087253
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/arktts_runtime/voices.py
@@ -0,0 +1,43 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import numpy as np
+
+
+class VoiceStore:
+ def __init__(self, root: Path, num_codebooks: int, codebook_size: int = 4096):
+ self.root = Path(root).resolve()
+ self.num_codebooks = int(num_codebooks)
+ self.codebook_size = int(codebook_size)
+
+ def list(self) -> list[dict]:
+ voices: list[dict] = []
+ if not self.root.exists():
+ return voices
+ for meta_path in sorted(self.root.glob("*/meta.json")):
+ try:
+ voices.append(json.loads(meta_path.read_text(encoding="utf-8")))
+ except (OSError, json.JSONDecodeError):
+ continue
+ return voices
+
+ def load(self, name: str) -> tuple[np.ndarray, dict]:
+ if not name or Path(name).name != name:
+ raise ValueError("invalid voice name")
+ voice_dir = self.root / name
+ meta_path = voice_dir / "meta.json"
+ codes_path = voice_dir / "codes.npy"
+ if not meta_path.is_file() or not codes_path.is_file():
+ raise KeyError(f"voice not found: {name}")
+ meta = json.loads(meta_path.read_text(encoding="utf-8"))
+ codes = np.load(codes_path, allow_pickle=False).astype(np.int64, copy=False)
+ if codes.ndim != 2 or codes.shape[0] != self.num_codebooks or codes.shape[1] == 0:
+ raise ValueError(f"invalid codes for voice {name}: {codes.shape}")
+ if int(codes.min()) < 0 or int(codes.max()) >= self.codebook_size:
+ raise ValueError(f"voice {name} contains code outside [0, {self.codebook_size})")
+ reference_text = str(meta.get("reference_text", "")).strip()
+ if not reference_text:
+ raise ValueError(f"voice {name} has no reference_text")
+ return codes, meta
diff --git a/onnx_runtime_0_1b_int8/requirements.txt b/onnx_runtime_0_1b_int8/requirements.txt
new file mode 100644
index 0000000..a2df4cd
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/requirements.txt
@@ -0,0 +1,10 @@
+# Python 3.10+ is supported by this standalone 0.1B runtime.
+numpy>=1.24,<3
+onnxruntime>=1.22,<1.24
+tokenizers>=0.20,<0.23
+soundfile>=0.12,<1
+scipy>=1.10,<2
+fastapi>=0.110,<1
+uvicorn>=0.29,<1
+pydantic>=2.6,<3
+python-multipart>=0.0.9,<1
diff --git a/onnx_runtime_0_1b_int8/run_infer.sh b/onnx_runtime_0_1b_int8/run_infer.sh
new file mode 100755
index 0000000..f652811
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/run_infer.sh
@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")" && pwd)"
+exec "$ROOT/.venv/bin/python" -m arktts_runtime.cli \
+ --model-dir "${ARKTTS_MODEL_DIR:-$ROOT/model}" \
+ --voices-dir "${ARKTTS_VOICES_DIR:-$ROOT/voices}" \
+ "$@"
diff --git a/onnx_runtime_0_1b_int8/run_server.sh b/onnx_runtime_0_1b_int8/run_server.sh
new file mode 100755
index 0000000..21ecde8
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/run_server.sh
@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")" && pwd)"
+export ARKTTS_MODEL_DIR="${ARKTTS_MODEL_DIR:-$ROOT/model}"
+export ARKTTS_VOICES_DIR="${ARKTTS_VOICES_DIR:-$ROOT/voices}"
+export ARKTTS_REGISTRATION_DIR="${ARKTTS_REGISTRATION_DIR:-$ARKTTS_MODEL_DIR/registration}"
+export ARKTTS_PRECISION="${ARKTTS_PRECISION:-int8}"
+export ARKTTS_CODEC_PRECISION="${ARKTTS_CODEC_PRECISION:-fp16}"
+export ARKTTS_THREADS="${ARKTTS_THREADS:-5}"
+
+exec "$ROOT/.venv/bin/uvicorn" arktts_runtime.service:app \
+ --app-dir "$ROOT" \
+ --host "${HOST:-127.0.0.1}" \
+ --port "${PORT:-8024}"
diff --git a/onnx_runtime_0_1b_int8/scripts/register_default_voice.py b/onnx_runtime_0_1b_int8/scripts/register_default_voice.py
new file mode 100755
index 0000000..2503f97
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/scripts/register_default_voice.py
@@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+"""Create the bundled ``default`` voice from the model's reference codes."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import tempfile
+from datetime import datetime, timezone
+from pathlib import Path
+
+import numpy as np
+
+
+def main() -> None:
+ root = Path(__file__).resolve().parents[1]
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--model-dir", type=Path, default=root / "model")
+ parser.add_argument("--voices-dir", type=Path, default=root / "voices")
+ parser.add_argument("--name", default="default")
+ parser.add_argument("--overwrite", action="store_true")
+ args = parser.parse_args()
+
+ manifest = json.loads((args.model_dir / "runtime_manifest.json").read_text(encoding="utf-8"))
+ codes_path = args.model_dir / manifest.get("reference_codes", "reference_codes.npy")
+ codes = np.load(codes_path, allow_pickle=False)
+ expected = int(manifest["num_codebooks"])
+ codebook_size = int(manifest.get("codebook_size", 4096))
+ if (
+ codes.ndim != 2
+ or codes.shape[0] != expected
+ or codes.shape[1] == 0
+ or int(codes.min()) < 0
+ or int(codes.max()) >= codebook_size
+ ):
+ raise ValueError(f"reference codes must have shape [{expected}, T] in [0, {codebook_size})")
+ name = str(args.name).strip()
+ if not name or Path(name).name != name or name in {".", ".."}:
+ raise ValueError("name must be one path component")
+ target = args.voices_dir / name
+ if target.exists() and not args.overwrite:
+ raise FileExistsError(f"voice already exists: {target}; pass --overwrite to replace it")
+
+ args.voices_dir.mkdir(parents=True, exist_ok=True)
+ temp = Path(tempfile.mkdtemp(prefix=f".{name}.", dir=args.voices_dir))
+ try:
+ np.save(temp / "codes.npy", codes.astype(np.uint16, copy=False))
+ meta = {
+ "name": name,
+ "reference_text": str(manifest["reference_text"]),
+ "shape": list(codes.shape),
+ "dtype": "uint16",
+ "sample_rate": int(manifest["sample_rate"]),
+ "model_fingerprint": manifest["model_fingerprint"],
+ "created_at": datetime.now(timezone.utc).isoformat(),
+ "source_kind": "model_reference_codes",
+ }
+ (temp / "meta.json").write_text(
+ json.dumps(meta, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ if target.exists():
+ import shutil
+
+ shutil.rmtree(target)
+ os.replace(temp, target)
+ finally:
+ if temp.exists():
+ import shutil
+
+ shutil.rmtree(temp)
+ print(f"registered {name}: shape={tuple(codes.shape)} at {target}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/onnx_runtime_0_1b_int8/setup.sh b/onnx_runtime_0_1b_int8/setup.sh
new file mode 100755
index 0000000..72cd5fb
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/setup.sh
@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")" && pwd)"
+PYTHON_BIN="${PYTHON_BIN:-python3}"
+
+if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
+ echo "Python was not found. Set PYTHON_BIN to a Python 3.10+ executable." >&2
+ exit 1
+fi
+if ! "$PYTHON_BIN" -c 'import sys; raise SystemExit(sys.version_info < (3, 10))'; then
+ echo "Python 3.10 or newer is required." >&2
+ exit 1
+fi
+
+"$PYTHON_BIN" -m venv "$ROOT/.venv"
+"$ROOT/.venv/bin/python" -m pip install --upgrade pip
+"$ROOT/.venv/bin/python" -m pip install -r "$ROOT/requirements.txt"
+echo "Environment ready: $ROOT/.venv"
diff --git a/onnx_runtime_0_1b_int8/start_server.sh b/onnx_runtime_0_1b_int8/start_server.sh
new file mode 100755
index 0000000..2c03aa2
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/start_server.sh
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")" && pwd)"
+PID_FILE="$ROOT/.service.pid"
+PORT="${PORT:-8024}"
+
+if [[ -f "$PID_FILE" ]] && kill -0 "$(<"$PID_FILE")" 2>/dev/null; then
+ echo "Service is already running with PID $(<"$PID_FILE")"
+ exit 0
+fi
+
+PORT="$PORT" nohup "$ROOT/run_server.sh" >"$ROOT/service.log" 2>&1 &
+PID=$!
+echo "$PID" >"$PID_FILE"
+for _ in {1..60}; do
+ if curl -fsS "http://127.0.0.1:$PORT/api/health" >/dev/null 2>&1; then
+ echo "Audio8 0.1B TTS is ready: http://127.0.0.1:$PORT"
+ exit 0
+ fi
+ if ! kill -0 "$PID" 2>/dev/null; then
+ echo "Service exited during startup. See $ROOT/service.log" >&2
+ exit 1
+ fi
+ sleep 1
+done
+echo "Service did not become ready in 60 seconds. See $ROOT/service.log" >&2
+exit 1
diff --git a/onnx_runtime_0_1b_int8/stop_server.sh b/onnx_runtime_0_1b_int8/stop_server.sh
new file mode 100755
index 0000000..cbf1c4f
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/stop_server.sh
@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")" && pwd)"
+PID_FILE="$ROOT/.service.pid"
+if [[ ! -f "$PID_FILE" ]]; then
+ echo "Service is not running"
+ exit 0
+fi
+PID="$(<"$PID_FILE")"
+if kill -0 "$PID" 2>/dev/null; then
+ kill "$PID"
+ echo "Stopped Audio8 0.1B TTS (PID $PID)"
+else
+ echo "Service PID $PID is not running"
+fi
+rm -f "$PID_FILE"
diff --git a/onnx_runtime_0_1b_int8/tests/test_contract.py b/onnx_runtime_0_1b_int8/tests/test_contract.py
new file mode 100644
index 0000000..80ac71b
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/tests/test_contract.py
@@ -0,0 +1,84 @@
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+import pytest
+
+ort = pytest.importorskip("onnxruntime")
+
+ROOT = Path(__file__).resolve().parents[1]
+MODEL_DIR = Path(os.getenv("ARKTTS_MODEL_DIR", str(ROOT / "model")))
+
+
+def _session(name: str):
+ path = MODEL_DIR / name
+ if not path.is_file():
+ pytest.skip(f"model file is not available: {path}")
+ return ort.InferenceSession(str(path), providers=["CPUExecutionProvider"])
+
+
+def test_slow_hybrid_contract() -> None:
+ session = _session("slow_ar_int8.onnx")
+ inputs = {item.name: item for item in session.get_inputs()}
+ assert set(inputs) == {
+ "codes",
+ "position",
+ "cache_keys",
+ "cache_values",
+ "conv_states",
+ "ssm_states",
+ }
+ assert tuple(inputs["codes"].shape) == (1, 11, 1)
+ assert tuple(inputs["position"].shape) == (1,)
+ assert tuple(inputs["cache_keys"].shape) == (24, 1, 2, 2048, 64)
+ assert tuple(inputs["cache_values"].shape) == (24, 1, 2, 2048, 64)
+ assert tuple(inputs["conv_states"].shape) == (24, 1, 896, 4)
+ assert tuple(inputs["ssm_states"].shape) == (24, 1, 24, 32, 64)
+
+ outputs = {item.name: item for item in session.get_outputs()}
+ assert set(outputs) == {
+ "logits",
+ "hidden",
+ "key_delta",
+ "value_delta",
+ "next_conv_states",
+ "next_ssm_states",
+ }
+ assert tuple(outputs["logits"].shape) == (1, 1, 4097)
+ assert tuple(outputs["hidden"].shape) == (1, 1, 512)
+
+
+def test_fast_and_decoder_contract() -> None:
+ fast = _session("fast_ar_int8.onnx")
+ fast_inputs = {item.name: item for item in fast.get_inputs()}
+ assert {name for name in fast_inputs if name.startswith("cache_key_")} == {
+ "cache_key_0",
+ "cache_key_1",
+ "cache_key_2",
+ "cache_key_3",
+ }
+ assert {name for name in fast_inputs if name.startswith("cache_value_")} == {
+ "cache_value_0",
+ "cache_value_1",
+ "cache_value_2",
+ "cache_value_3",
+ }
+ assert tuple(fast_inputs["slow_hidden"].shape) == (1, 1, 512)
+ assert tuple(fast_inputs["token_id"].shape) == (1, 1)
+ assert tuple(fast_inputs["input_pos"].shape) == (1,)
+ assert [item.name for item in fast.get_outputs()] == [
+ "logits",
+ "key_delta_0",
+ "value_delta_0",
+ "key_delta_1",
+ "value_delta_1",
+ "key_delta_2",
+ "value_delta_2",
+ "key_delta_3",
+ "value_delta_3",
+ ]
+ assert tuple(fast.get_outputs()[0].shape) == (1, 1, 4096)
+
+ decoder = _session("codec_decoder_fp16.onnx")
+ assert tuple(decoder.get_inputs()[0].shape[1:2]) == (10,)
diff --git a/onnx_runtime_0_1b_int8/tests/test_prompt.py b/onnx_runtime_0_1b_int8/tests/test_prompt.py
new file mode 100644
index 0000000..c81bea5
--- /dev/null
+++ b/onnx_runtime_0_1b_int8/tests/test_prompt.py
@@ -0,0 +1,27 @@
+from __future__ import annotations
+
+import numpy as np
+import pytest
+
+pytest.importorskip("tokenizers")
+
+from arktts_runtime.prompt import clean_text, format_reference_text
+
+
+def test_text_normalization_preserves_special_tokens() -> None:
+ assert clean_text("\x00 你好\n世界 ") == "你好世界"
+ assert clean_text("你好\nworld") == "你好 world"
+ assert format_reference_text("你好") == "<|speaker:0|>你好"
+ assert format_reference_text("<|speaker:2|>hello") == "<|speaker:2|>hello"
+
+
+def test_sample_state_update_handles_full_and_delta_cache() -> None:
+ from arktts_runtime.runtime import ArkTtsRuntime
+
+ cache = np.zeros((1, 2, 10, 64), dtype=np.float32)
+ delta = np.ones_like(cache[:, :, :1])
+ ArkTtsRuntime._update_fast_cache(cache, delta, 3)
+ assert np.all(cache[:, :, 3:4] == 1)
+ replacement = np.full_like(cache, 2)
+ ArkTtsRuntime._update_fast_cache(cache, replacement, 0)
+ assert np.all(cache == 2)