SOTA-Class TTS at Compact Scale
Python
Add standalone 0.1B INT8 ONNX runtime
chuarunyuan4e8e99fe9286
diff --git a/onnx_runtime_0_1b_int8/README.md b/onnx_runtime_0_1b_int8/README.mdnew file mode 100644index 0000000..6836efe--- /dev/null+++ b/onnx_runtime_0_1b_int8/README.md@@ -0,0 +1,99 @@+# Audio8 0.1B INT8 ONNX Runtime++This directory is a self-contained runtime for the+`Audio8/audio8-TTS-0.1B-ONNX-INT8` export. It is intentionally separate from+the existing `onnx_runtime/` implementation, which targets the incompatible+0.6B INT4 graph.++## Why this runtime is separate++The 0.1B checkpoint uses a Falcon-H1 hybrid slow graph. The graph accepts one+`[1, 11, 1]` token column and carries four recurrent tensors between calls:++```text+cache_keys [24, 1, 2, 2048, 64]+cache_values [24, 1, 2, 2048, 64]+conv_states [24, 1, 896, 4]+ssm_states [24, 1, 24, 32, 64]+```++The implementation below performs prompt prefill one position at a time,+updates the returned state deltas, maps the compact 4097-way semantic logits,+then runs the four-layer Fast AR codebook graph and the FP16 codec decoder.++## Install++Python 3.10 or newer is supported, including the Python 3.10 environment used+by the Jetson issue report. From this directory:++```bash+python3 -m pip install -U "huggingface_hub[cli]"+hf download Audio8/audio8-TTS-0.1B-ONNX-INT8 --local-dir model+bash setup.sh+python scripts/register_default_voice.py+```++The model directory must contain:++```text+model/+|- slow_ar_int8.onnx(.data)+|- fast_ar_int8.onnx(.data)+|- codec_decoder_fp16.onnx(.data)+|- runtime_manifest.json+|- reference_codes.npy+|- tokenizer/tokenizer.json+`- registration/+ |- codec_encoder_fp16.onnx(.data) (optional)+ `- registration_manifest.json (optional)+```++`register_default_voice.py` only uses `reference_codes.npy` and does not load+the optional encoder. To register a new voice from audio, use the service API+after the optional `registration/` files have been downloaded.++## CLI++```bash+bash run_infer.sh \+ --text "这是一个中文测试" \+ --voice default \+ --max-new-tokens 128 \+ --output outputs/test.wav+```++The command also writes `outputs/test.npy` with generated codes shaped+`[10, frames]`.++Set `ARKTTS_MODEL_DIR` or `ARKTTS_VOICES_DIR` to use another location. The+runtime always selects `CPUExecutionProvider`; GPU execution is outside the+scope of this export.++## Local HTTP service++```bash+bash start_server.sh+curl http://127.0.0.1:8024/api/health+curl http://127.0.0.1:8024/api/tts \+ -H 'Content-Type: application/json' \+ -d '{"text":"你好,这是一个测试。","voice_name":"default"}' \+ -o outputs/api.wav+```++The service exposes `/api/tts`, `/api/tts/stream`, `/api/tts/cancel`,+`/api/voices`, `/api/voices/register`, `/api/registration/status`, and the+OpenAI-compatible `/v1/audio/speech`. Stop it with `bash stop_server.sh`.++## Verification++```bash+"$PWD/.venv/bin/python" -m pytest -q tests+```++`test_contract.py` checks the exact input/output names and shapes when+`ARKTTS_MODEL_DIR` points at a downloaded model. Without model files it skips+the ONNX-dependent check; the pure prompt and state-shape tests still run.++The 0.1B model and this runtime are intended for local, authorized voice+generation. Obtain consent before cloning a voice and disclose synthetic audio+where appropriate.diff --git a/onnx_runtime_0_1b_int8/README_zh.md b/onnx_runtime_0_1b_int8/README_zh.mdnew file mode 100644index 0000000..56d1f10--- /dev/null+++ b/onnx_runtime_0_1b_int8/README_zh.md@@ -0,0 +1,62 @@+# Audio8 0.1B INT8 ONNX Runtime++这是 `Audio8/audio8-TTS-0.1B-ONNX-INT8` 的独立 CPU ONNX Runtime。它放在+单独目录中,因为仓库现有的 `onnx_runtime/` 是给 0.6B INT4 图使用的,不能+直接处理 0.1B Falcon-H1 hybrid 图。++0.1B Slow AR 每次只接受一个 `[1, 11, 1]` token column,并且要在调用之间保存:++```text+cache_keys [24, 1, 2, 2048, 64]+cache_values [24, 1, 2, 2048, 64]+conv_states [24, 1, 896, 4]+ssm_states [24, 1, 24, 32, 64]+```++本实现会逐位置执行 prompt prefill,写回 Slow AR 返回的 state delta,处理+4097 维 compact semantic logits,然后调用四层 Fast AR 和 FP16 codec decoder。++## 安装++支持 Python 3.10 及以上版本,Jetson issue 中的 Python 3.10 也可以使用:++```bash+python3 -m pip install -U "huggingface_hub[cli]"+hf download Audio8/audio8-TTS-0.1B-ONNX-INT8 --local-dir model+bash setup.sh+python scripts/register_default_voice.py+```++模型文件放在本目录的 `model/` 下。`registration/` 下的 encoder 是可选的,+只在从音频注册新音色时需要。默认音色脚本只读取 `reference_codes.npy`。++## 命令行推理++```bash+bash run_infer.sh \+ --text "这是一个中文测试" \+ --voice default \+ --max-new-tokens 128 \+ --output outputs/test.wav+```++同时会生成 `[10, frames]` 形状的 `outputs/test.npy`。模型或音色放在其他位置时,+可设置 `ARKTTS_MODEL_DIR`、`ARKTTS_VOICES_DIR`。++## HTTP 服务++```bash+bash start_server.sh+curl http://127.0.0.1:8024/api/health+```++服务提供 `/api/tts`、`/api/tts/stream`、`/api/tts/cancel`、音色查询和注册接口,+以及兼容 OpenAI 的 `/v1/audio/speech`。使用 `bash stop_server.sh` 停止服务。++## 测试++```bash+"$PWD/.venv/bin/python" -m pytest -q tests+```++请在获得授权后进行音色克隆,并在适当场景披露合成音频。diff --git a/onnx_runtime_0_1b_int8/arktts_runtime/__init__.py b/onnx_runtime_0_1b_int8/arktts_runtime/__init__.pynew file mode 100644index 0000000..622a289--- /dev/null+++ b/onnx_runtime_0_1b_int8/arktts_runtime/__init__.py@@ -0,0 +1,5 @@+"""Standalone ONNX Runtime for Audio8 0.1B INT8 exports."""++from .runtime import ArkTtsRuntime++__all__ = ["ArkTtsRuntime"]diff --git a/onnx_runtime_0_1b_int8/arktts_runtime/cli.py b/onnx_runtime_0_1b_int8/arktts_runtime/cli.pynew file mode 100644index 0000000..bdf9843--- /dev/null+++ b/onnx_runtime_0_1b_int8/arktts_runtime/cli.py@@ -0,0 +1,55 @@+from __future__ import annotations++import argparse+from pathlib import Path++import numpy as np+import soundfile as sf++from .runtime import ArkTtsRuntime++ROOT = Path(__file__).resolve().parents[1]+++def main() -> None:+ parser = argparse.ArgumentParser(description="Audio8 0.1B INT8 ONNX Runtime")+ parser.add_argument("--model-dir", type=Path, default=ROOT / "model")+ parser.add_argument("--voices-dir", type=Path, default=ROOT / "voices")+ parser.add_argument("--precision", choices=["int8"], default=None)+ parser.add_argument("--codec-precision", choices=["fp16"], default=None)+ parser.add_argument("--text", required=True)+ parser.add_argument("--voice", default="default")+ parser.add_argument("--output", type=Path, default=Path("output.wav"))+ parser.add_argument("--max-new-tokens", type=int, default=256)+ parser.add_argument("--temperature", type=float, default=0.7)+ parser.add_argument("--top-p", type=float, default=0.9)+ parser.add_argument("--top-k", type=int, default=50)+ parser.add_argument("--seed", type=int, default=42)+ parser.add_argument("--threads", type=int, default=5)+ args = parser.parse_args()++ runtime = ArkTtsRuntime(+ args.model_dir,+ args.voices_dir,+ args.precision,+ args.codec_precision,+ args.threads,+ )+ audio, codes = runtime.synthesize(+ text=args.text,+ voice=args.voice,+ max_new_tokens=args.max_new_tokens,+ temperature=args.temperature,+ top_p=args.top_p,+ top_k=args.top_k,+ seed=args.seed,+ )+ args.output.parent.mkdir(parents=True, exist_ok=True)+ sf.write(str(args.output), audio, int(runtime.manifest["sample_rate"]))+ np.save(args.output.with_suffix(".npy"), codes.astype(np.uint16))+ print(f"saved {args.output}")+ print(f"saved {args.output.with_suffix('.npy')}")+++if __name__ == "__main__":+ main()diff --git a/onnx_runtime_0_1b_int8/arktts_runtime/prompt.py b/onnx_runtime_0_1b_int8/arktts_runtime/prompt.pynew file mode 100644index 0000000..37ca2a5--- /dev/null+++ b/onnx_runtime_0_1b_int8/arktts_runtime/prompt.py@@ -0,0 +1,93 @@+from __future__ import annotations++import re+import unicodedata+from pathlib import Path++import numpy as np+from tokenizers import Tokenizer++_CJK_RANGES = (+ "\u1100-\u11ff\u2e80-\u2fdf\u3000-\u303f\u3040-\u30ff\u3100-\u31ff"+ "\u3400-\u4dbf\u4e00-\u9fff\ua960-\ua97f\uac00-\ud7a3\ud7b0-\ud7ff\uf900-\ufaff"+ "\ufe30-\ufe4f\uff01-\uff9f\U00020000-\U0002fa1f"+)+_CJK_CHARACTER_RE = re.compile(rf"[{_CJK_RANGES}]")+_LINE_BREAK_RE = re.compile(r"[\r\n\v\f\x1c-\x1e\x85\u2028\u2029]")+++def _normalize_whitespace(text: str) -> str:+ def replace(match: re.Match[str]) -> str:+ left = text[match.start() - 1] if match.start() else ""+ right = text[match.end()] if match.end() < len(text) else ""+ if (+ _LINE_BREAK_RE.search(match.group())+ and _CJK_CHARACTER_RE.fullmatch(left)+ and _CJK_CHARACTER_RE.fullmatch(right)+ ):+ return ""+ return " "++ return re.sub(r"\s+", replace, text).strip()+++def clean_text(text: str) -> str:+ value = "".join(+ char+ if char.isspace()+ else ""+ if unicodedata.category(char).startswith("C")+ else char+ for char in str(text)+ )+ return _normalize_whitespace(value)+++def format_reference_text(text: str) -> str:+ text = clean_text(text)+ return text if re.search(r"<\|speaker:\d+\|>", text) else f"<|speaker:0|>{text}"+++class PromptBuilder:+ def __init__(self, tokenizer_dir: Path, semantic_begin_id: int, num_codebooks: int):+ self.tokenizer = Tokenizer.from_file(str(Path(tokenizer_dir) / "tokenizer.json"))+ self.semantic_begin_id = int(semantic_begin_id)+ self.num_codebooks = int(num_codebooks)++ def encode_text(self, text: str) -> list[int]:+ return list(self.tokenizer.encode(text, add_special_tokens=False).ids)++ def build(self, target_text: str, reference_text: str, reference_codes: np.ndarray) -> np.ndarray:+ target = clean_text(target_text)+ if not target:+ raise ValueError("text must not be empty")+ codes = np.asarray(reference_codes, dtype=np.int64)+ if codes.ndim != 2 or codes.shape[0] != self.num_codebooks or codes.shape[1] == 0:+ raise ValueError(+ f"reference codes must have shape [{self.num_codebooks}, T>0], got {codes.shape}"+ )+ if int(codes.min()) < 0:+ raise ValueError("reference codes must be non-negative")++ prefix_parts = [+ "<|im_start|>system\n",+ "convert the provided text to speech reference to the following:\n\nText:\n",+ format_reference_text(reference_text),+ "\n\nSpeech:\n",+ ]+ suffix_parts = [+ "<|im_end|>\n",+ "<|im_start|>user\n",+ target,+ "<|im_end|>\n",+ "<|im_start|>assistant\n<|voice|>",+ ]+ prefix = [token for part in prefix_parts for token in self.encode_text(part)]+ suffix = [token for part in suffix_parts for token in self.encode_text(part)]+ semantic_ids = (codes[0] + self.semantic_begin_id).tolist()+ row0 = np.asarray(prefix + semantic_ids + suffix, dtype=np.int64)+ values = np.zeros((self.num_codebooks + 1, row0.size), dtype=np.int64)+ values[0] = row0+ reference_start = len(prefix)+ values[1:, reference_start : reference_start + codes.shape[1]] = codes+ return values[np.newaxis]diff --git a/onnx_runtime_0_1b_int8/arktts_runtime/registration.py b/onnx_runtime_0_1b_int8/arktts_runtime/registration.pynew file mode 100644index 0000000..10c5f23--- /dev/null+++ b/onnx_runtime_0_1b_int8/arktts_runtime/registration.py@@ -0,0 +1,162 @@+from __future__ import annotations++import gc+import hashlib+import io+import json+import os+import shutil+import tempfile+from datetime import datetime, timezone+from math import gcd+from pathlib import Path++import numpy as np+import onnxruntime as ort+import soundfile as sf++MAX_AUDIO_BYTES = 50 * 1024 * 1024+MAX_AUDIO_SECONDS = 30.0+MIN_AUDIO_SECONDS = 0.5+++class VoiceRegistration:+ """Optional reference-audio encoder used to create a voice profile."""++ def __init__(self, registration_dir: Path, voices_root: Path, model_fingerprint: str):+ self.root = Path(registration_dir).resolve()+ self.voices_root = Path(voices_root).resolve()+ self.encoder_path = self.root / "codec_encoder_fp16.onnx"+ self.manifest_path = self.root / "registration_manifest.json"+ self.expected_fingerprint = model_fingerprint++ def status(self) -> dict[str, str | bool | None]:+ available = self.encoder_path.is_file() and self.manifest_path.is_file()+ reason: str | None = None+ if available:+ try:+ manifest = json.loads(self.manifest_path.read_text(encoding="utf-8"))+ if manifest.get("model_fingerprint") != self.expected_fingerprint:+ available = False+ reason = "registration model fingerprint does not match online model"+ except (OSError, json.JSONDecodeError):+ available = False+ reason = "registration manifest is invalid"+ else:+ reason = "registration package was not found"+ return {"available": available, "reason": reason}++ @staticmethod+ def _validate_name(name: str) -> str:+ value = name.strip()+ if not value or len(value) > 64 or value in {".", ".."} or Path(value).name != value:+ raise ValueError("voice name must be one path component with at most 64 characters")+ return value++ @staticmethod+ def _decode_audio(data: bytes, target_rate: int) -> tuple[np.ndarray, int]:+ if not data or len(data) > MAX_AUDIO_BYTES:+ raise ValueError("audio file must be between 1 byte and 50 MiB")+ try:+ audio, sample_rate = sf.read(io.BytesIO(data), dtype="float32", always_2d=True)+ except (RuntimeError, sf.LibsndfileError) as exc:+ raise ValueError(f"unsupported or invalid audio file: {exc}") from exc+ audio = audio.mean(axis=1)+ source_rate = int(sample_rate)+ duration = audio.size / max(1, source_rate)+ if duration < MIN_AUDIO_SECONDS or duration > MAX_AUDIO_SECONDS:+ raise ValueError("reference audio duration must be between 0.5 and 30 seconds")+ if not np.isfinite(audio).all():+ raise ValueError("reference audio contains non-finite samples")+ if source_rate != int(target_rate):+ from scipy.signal import resample_poly++ factor = gcd(source_rate, int(target_rate))+ audio = resample_poly(+ audio,+ int(target_rate) // factor,+ source_rate // factor,+ ).astype(np.float32)+ padding = (-audio.size) % 2048+ if padding:+ audio = np.pad(audio, (0, padding))+ return np.ascontiguousarray(audio.reshape(1, 1, -1)), source_rate++ def register(+ self,+ data: bytes,+ filename: str,+ text: str,+ name: str,+ overwrite: bool = False,+ ) -> dict:+ state = self.status()+ if not state["available"]:+ raise RuntimeError(str(state["reason"]))+ voice_name = self._validate_name(name)+ reference_text = " ".join(text.strip().split())+ if not reference_text:+ raise ValueError("reference text must not be empty")+ manifest = json.loads(self.manifest_path.read_text(encoding="utf-8"))+ audio, source_rate = self._decode_audio(data, int(manifest["sample_rate"]))+ target = self.voices_root / voice_name+ if target.exists() and not overwrite:+ raise FileExistsError(f"voice already exists: {voice_name}")++ options = ort.SessionOptions()+ options.log_severity_level = 3+ session = None+ try:+ session = ort.InferenceSession(+ str(self.encoder_path),+ sess_options=options,+ providers=["CPUExecutionProvider"],+ )+ input_info = session.get_inputs()[0]+ input_type = np.float16 if input_info.type == "tensor(float16)" else np.float32+ codes = np.asarray(session.run(None, {input_info.name: audio.astype(input_type)})[0])+ finally:+ del session+ gc.collect()++ if codes.ndim == 3:+ codes = codes[0]+ expected_codebooks = int(manifest["num_codebooks"])+ codebook_size = int(manifest.get("codebook_size", 4096))+ if (+ codes.ndim != 2+ or codes.shape[0] != expected_codebooks+ or codes.shape[1] == 0+ or int(codes.min()) < 0+ or int(codes.max()) >= codebook_size+ ):+ raise RuntimeError(f"encoder returned invalid codes: {codes.shape}")++ self.voices_root.mkdir(parents=True, exist_ok=True)+ temp = Path(tempfile.mkdtemp(prefix=f".{voice_name}.", dir=self.voices_root))+ try:+ np.save(temp / "codes.npy", codes.astype(np.uint16))+ meta = {+ "name": voice_name,+ "reference_text": reference_text,+ "shape": list(codes.shape),+ "dtype": "uint16",+ "sample_rate": int(manifest["sample_rate"]),+ "source_audio": Path(filename or "reference_audio").name,+ "source_sample_rate": source_rate,+ "source_sha256": hashlib.sha256(data).hexdigest(),+ "model_fingerprint": manifest["model_fingerprint"],+ "created_at": datetime.now(timezone.utc).isoformat(),+ "source_kind": "onnx_runtime_registration",+ }+ (temp / "meta.json").write_text(+ json.dumps(meta, ensure_ascii=False, indent=2) + "\n",+ encoding="utf-8",+ )+ if target.exists():+ shutil.rmtree(target)+ os.replace(temp, target)+ finally:+ if temp.exists():+ shutil.rmtree(temp)+ return metadiff --git a/onnx_runtime_0_1b_int8/arktts_runtime/runtime.py b/onnx_runtime_0_1b_int8/arktts_runtime/runtime.pynew file mode 100644index 0000000..c9b2952--- /dev/null+++ b/onnx_runtime_0_1b_int8/arktts_runtime/runtime.py@@ -0,0 +1,481 @@+from __future__ import annotations++import json+import threading+from collections.abc import Iterator+from pathlib import Path+from typing import Any++import numpy as np+import onnxruntime as ort++from .prompt import PromptBuilder+from .voices import VoiceStore+++ORT_DTYPES: dict[str, Any] = {+ "tensor(float)": np.float32,+ "tensor(float16)": np.float16,+ "tensor(int64)": np.int64,+ "tensor(int32)": np.int32,+ "tensor(bool)": np.bool_,+}+++def _session(path: Path, threads: int | None = None) -> ort.InferenceSession:+ if not path.is_file():+ raise FileNotFoundError(f"ONNX model was not found: {path}")+ options = ort.SessionOptions()+ options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL+ options.log_severity_level = 3+ if threads is not None:+ options.intra_op_num_threads = max(1, int(threads))+ options.inter_op_num_threads = max(1, int(threads) // 2)+ return ort.InferenceSession(+ str(path),+ sess_options=options,+ providers=["CPUExecutionProvider"],+ )+++def _sample(+ logits: np.ndarray,+ temperature: float,+ top_p: float,+ top_k: int,+ rng: np.random.Generator,+) -> int:+ values = np.asarray(logits, dtype=np.float64).reshape(-1)+ if values.size == 0 or not np.isfinite(values).any():+ raise ValueError("model returned empty or non-finite logits")+ temperature = max(float(temperature), 1e-5)+ top_p = min(max(float(top_p), 1e-5), 1.0)+ top_k = min(max(int(top_k), 1), values.size)++ order = np.argsort(values)[::-1]+ sorted_values = values[order]+ probabilities = np.exp(sorted_values - np.max(sorted_values))+ probabilities /= probabilities.sum()+ cumulative = np.cumsum(probabilities)+ remove = (cumulative > top_p) | (np.arange(values.size) >= top_k)+ remove[0] = False+ filtered = values.copy()+ filtered[order[remove]] = -np.inf+ scaled = filtered / temperature+ scaled -= np.max(scaled)+ probabilities = np.exp(scaled)+ probabilities /= probabilities.sum()+ # Gumbel-max is stable and makes the RNG sequence explicit for reproducible CLI runs.+ uniform = np.clip(rng.random(probabilities.size), 1e-12, 1.0 - 1e-12)+ gumbel = -np.log(-np.log(uniform))+ return int(np.argmax(np.log(np.clip(probabilities, 1e-300, 1.0)) + gumbel))+++def _dtype(value: str) -> np.dtype:+ try:+ return np.dtype(ORT_DTYPES[value])+ except KeyError as exc:+ raise ValueError(f"unsupported ONNX tensor type: {value}") from exc+++def _shape(info: ort.NodeArg, fallback: tuple[int, ...]) -> tuple[int, ...]:+ """Resolve symbolic dimensions to the dimensions used by this export."""+ result: list[int] = []+ for index, dimension in enumerate(info.shape):+ if isinstance(dimension, int) and dimension > 0:+ result.append(dimension)+ elif index < len(fallback):+ result.append(int(fallback[index]))+ else:+ raise ValueError(f"cannot resolve dynamic shape for {info.name}: {info.shape}")+ return tuple(result)+++class ArkTtsRuntime:+ """CPU runtime for ``Audio8/audio8-TTS-0.1B-ONNX-INT8``.++ The 0.1B graph is a Falcon-H1 hybrid export. Its slow graph consumes one+ ``[1, 11, 1]`` column at a time and returns state deltas; it is not+ compatible with the 0.6B runtime's independent per-layer KV inputs.+ """++ _SLOW_INPUTS = {+ "codes",+ "position",+ "cache_keys",+ "cache_values",+ "conv_states",+ "ssm_states",+ }+ _FAST_INPUTS = {+ "slow_hidden",+ "token_id",+ "use_slow_hidden",+ "input_pos",+ "cache_key_0",+ "cache_key_1",+ "cache_key_2",+ "cache_key_3",+ "cache_value_0",+ "cache_value_1",+ "cache_value_2",+ "cache_value_3",+ }++ def __init__(+ self,+ model_dir: Path,+ voices_dir: Path,+ precision: str | None = None,+ codec_precision: str | None = None,+ threads: int | None = None,+ ):+ self.model_dir = Path(model_dir).resolve()+ manifest_path = self.model_dir / "runtime_manifest.json"+ if not manifest_path.is_file():+ raise FileNotFoundError(f"runtime manifest was not found: {manifest_path}")+ self.manifest = json.loads(manifest_path.read_text(encoding="utf-8"))+ self.precision = precision or self.manifest["default_precision"]+ if self.precision not in self.manifest["available_precisions"]:+ raise ValueError(f"unsupported precision: {self.precision}")+ self.codec_precision = codec_precision or self.manifest.get(+ "default_codec_precision", "fp16"+ )+ available_codec = self.manifest.get("available_codec_precisions", ["fp16"])+ if self.codec_precision not in available_codec:+ raise ValueError(f"unsupported codec precision: {self.codec_precision}")++ slow_name = self.manifest.get("slow_decode_models", {}).get(+ self.precision,+ self.manifest.get("slow_decode_model", f"slow_ar_{self.precision}.onnx"),+ )+ fast_name = self.manifest.get("fast_models", {}).get(+ self.precision,+ self.manifest.get("fast_model", f"fast_ar_{self.precision}.onnx"),+ )+ codec_name = self.manifest.get("codec_models", {}).get(+ self.codec_precision,+ f"codec_decoder_{self.codec_precision}.onnx",+ )+ self.slow = _session(self.model_dir / slow_name, threads)+ self.fast = _session(self.model_dir / fast_name, threads)+ self.decoder = _session(self.model_dir / codec_name, threads)++ self.slow_inputs = {item.name: item for item in self.slow.get_inputs()}+ self.fast_inputs = {item.name: item for item in self.fast.get_inputs()}+ self._validate_contract()+ self._slow_outputs = {item.name: index for index, item in enumerate(self.slow.get_outputs())}+ self._fast_outputs = {item.name: index for index, item in enumerate(self.fast.get_outputs())}++ self.prompt_builder = PromptBuilder(+ self.model_dir / "tokenizer",+ int(self.manifest["semantic_begin_id"]),+ int(self.manifest["num_codebooks"]),+ )+ self.voices = VoiceStore(Path(voices_dir), int(self.manifest["num_codebooks"]))++ def _validate_contract(self) -> None:+ slow_names = set(self.slow_inputs)+ missing = self._SLOW_INPUTS - slow_names+ if missing:+ raise ValueError(+ "this runtime expects the 0.1B hybrid slow graph; "+ f"missing inputs: {sorted(missing)}"+ )+ fast_missing = self._FAST_INPUTS - set(self.fast_inputs)+ if fast_missing:+ raise ValueError(f"0.1B fast graph is missing inputs: {sorted(fast_missing)}")+ if tuple(self.slow_inputs["codes"].shape[-2:]) != (11, 1):+ raise ValueError(f"unexpected slow codes shape: {self.slow_inputs['codes'].shape}")+ if tuple(self.slow_inputs["position"].shape) not in ((1,), (None,)):+ raise ValueError(f"unexpected slow position shape: {self.slow_inputs['position'].shape}")+ if int(self.manifest["num_layers"]) != int(self.slow_inputs["cache_keys"].shape[0]):+ raise ValueError("runtime_manifest num_layers does not match slow cache_keys")+ if int(self.manifest["num_fast_layers"]) != 4:+ raise ValueError("the 0.1B fast export must have four cache layers")++ def _empty_slow_state(self) -> dict[str, np.ndarray]:+ specs = {+ "cache_keys": (+ int(self.manifest["num_layers"]),+ 1,+ int(self.manifest["n_local_heads"]),+ int(self.manifest["max_seq_len"]),+ int(self.manifest["head_dim"]),+ ),+ "cache_values": (+ int(self.manifest["num_layers"]),+ 1,+ int(self.manifest["n_local_heads"]),+ int(self.manifest["max_seq_len"]),+ int(self.manifest["head_dim"]),+ ),+ "conv_states": (+ int(self.manifest["num_layers"]),+ 1,+ 896,+ int(self.manifest["mamba_d_conv"]),+ ),+ "ssm_states": (+ int(self.manifest["num_layers"]),+ 1,+ int(self.manifest["mamba_n_heads"]),+ int(self.manifest["mamba_d_head"]),+ int(self.manifest["mamba_d_state"]),+ ),+ }+ return {+ name: np.zeros(+ _shape(self.slow_inputs[name], fallback),+ dtype=_dtype(self.slow_inputs[name].type),+ )+ for name, fallback in specs.items()+ }++ def _empty_fast_caches(self) -> dict[str, np.ndarray]:+ result: dict[str, np.ndarray] = {}+ for name, info in self.fast_inputs.items():+ if not (name.startswith("cache_key_") or name.startswith("cache_value_")):+ continue+ fallback = (1, 2, int(self.manifest["num_codebooks"]), int(self.manifest["fast_head_dim"]))+ result[name] = np.zeros(_shape(info, fallback), dtype=_dtype(info.type))+ return result++ @staticmethod+ def _output(outputs: list[np.ndarray], names: dict[str, int], name: str, index: int) -> np.ndarray:+ output_index = names.get(name, index)+ if output_index >= len(outputs):+ raise ValueError(f"ONNX graph did not return expected output {name}")+ return np.asarray(outputs[output_index])++ def _slow_step(+ self,+ codes: np.ndarray,+ position: int,+ state: dict[str, np.ndarray],+ ) -> tuple[np.ndarray, np.ndarray]:+ codes = np.asarray(codes, dtype=np.int64)+ if codes.shape != (1, int(self.manifest["num_codebooks"]) + 1, 1):+ raise ValueError(f"slow codes must have shape [1, 11, 1], got {codes.shape}")+ position_value = np.asarray([int(position)], dtype=np.int64)+ feeds = {+ "codes": codes,+ "position": position_value,+ "cache_keys": state["cache_keys"],+ "cache_values": state["cache_values"],+ "conv_states": state["conv_states"],+ "ssm_states": state["ssm_states"],+ }+ outputs = self.slow.run(None, feeds)+ key_delta = self._output(outputs, self._slow_outputs, "key_delta", 2)+ value_delta = self._output(outputs, self._slow_outputs, "value_delta", 3)+ next_conv = self._output(outputs, self._slow_outputs, "next_conv_states", 4)+ next_ssm = self._output(outputs, self._slow_outputs, "next_ssm_states", 5)+ if not 0 <= int(position) < state["cache_keys"].shape[3]:+ raise ValueError(f"slow position {position} exceeds cache length")+ state["cache_keys"][:, :, :, int(position), :] = key_delta+ state["cache_values"][:, :, :, int(position), :] = value_delta+ state["conv_states"] = next_conv.astype(state["conv_states"].dtype, copy=False)+ state["ssm_states"] = next_ssm.astype(state["ssm_states"].dtype, copy=False)+ logits = self._output(outputs, self._slow_outputs, "logits", 0)+ hidden = self._output(outputs, self._slow_outputs, "hidden", 1)+ return logits[0, -1], hidden[:, -1:, :]++ def _fast_step(+ self,+ hidden: np.ndarray,+ token_id: int,+ use_slow_hidden: bool,+ position: int,+ caches: dict[str, np.ndarray],+ ) -> np.ndarray:+ hidden_info = self.fast_inputs["slow_hidden"]+ use_info = self.fast_inputs["use_slow_hidden"]+ feeds: dict[str, np.ndarray] = {+ "slow_hidden": np.asarray(hidden, dtype=_dtype(hidden_info.type)),+ "token_id": np.asarray([[int(token_id)]], dtype=_dtype(self.fast_inputs["token_id"].type)),+ "use_slow_hidden": np.asarray(+ [bool(use_slow_hidden)], dtype=_dtype(use_info.type)+ ),+ "input_pos": np.asarray([int(position)], dtype=_dtype(self.fast_inputs["input_pos"].type)),+ }+ feeds.update(caches)+ outputs = self.fast.run(None, feeds)+ for index in range(4):+ key_name = f"cache_key_{index}"+ value_name = f"cache_value_{index}"+ key_delta = self._output(+ outputs,+ self._fast_outputs,+ f"key_delta_{index}",+ 1 + index * 2,+ )+ value_delta = self._output(+ outputs,+ self._fast_outputs,+ f"value_delta_{index}",+ 2 + index * 2,+ )+ self._update_fast_cache(caches[key_name], key_delta, position)+ self._update_fast_cache(caches[value_name], value_delta, position)+ logits = self._output(outputs, self._fast_outputs, "logits", 0)+ return logits[0, -1]++ @staticmethod+ def _update_fast_cache(cache: np.ndarray, value: np.ndarray, position: int) -> None:+ value = np.asarray(value, dtype=cache.dtype)+ if value.shape == cache.shape:+ cache[...] = value+ return+ if value.ndim == cache.ndim and value.shape[:2] == cache.shape[:2] and value.shape[3:] == cache.shape[3:]:+ length = value.shape[2]+ cache[:, :, int(position) : int(position) + length, :] = value+ return+ raise ValueError(f"unexpected fast cache output shape {value.shape}; expected {cache.shape}")++ def _sample_semantic(+ self,+ logits: np.ndarray,+ previous: list[int],+ temperature: float,+ top_p: float,+ top_k: int,+ rng: np.random.Generator,+ ) -> int:+ begin = int(self.manifest["semantic_begin_id"])+ end = int(self.manifest["semantic_end_id"])+ stop = int(self.manifest["im_end_id"])+ expected = int(self.manifest["codebook_size"]) + 1+ values = np.asarray(logits, dtype=np.float32).reshape(-1)+ if self.manifest.get("slow_logits_layout") != "relative_semantic_then_eos":+ raise ValueError("this runtime only supports relative_semantic_then_eos logits")+ if values.size != expected:+ raise ValueError(f"unexpected slow logits size: {values.size}, expected {expected}")+ normal_index = _sample(values, temperature, top_p, top_k, rng)+ high_index = _sample(values, 1.0, 0.9, top_k, rng)+ normal = stop if normal_index == expected - 1 else begin + normal_index+ high = stop if high_index == expected - 1 else begin + high_index+ if begin <= normal <= end and normal in previous:+ return high+ return normal++ def iter_codes(+ self,+ text: str,+ voice: str,+ max_new_tokens: int = 1024,+ temperature: float = 0.7,+ top_p: float = 0.9,+ top_k: int = 50,+ seed: int = 42,+ stop_event: threading.Event | None = None,+ ) -> Iterator[np.ndarray]:+ reference_codes, meta = self.voices.load(voice)+ prompt = self.prompt_builder.build(text, meta["reference_text"], reference_codes)+ prompt_len = int(prompt.shape[2])+ max_seq_len = int(self.manifest["max_seq_len"])+ if prompt_len >= max_seq_len:+ raise ValueError(f"prompt length {prompt_len} exceeds max sequence length {max_seq_len}")+ max_new_tokens = min(max(0, int(max_new_tokens)), max_seq_len - prompt_len)++ rng = np.random.default_rng(int(seed))+ slow_state = self._empty_slow_state()+ logits: np.ndarray | None = None+ hidden: np.ndarray | None = None+ # The exported hybrid graph is a one-token graph, including prefill.+ for position in range(prompt_len):+ logits, hidden = self._slow_step(prompt[:, :, position : position + 1], position, slow_state)++ previous: list[int] = []+ begin = int(self.manifest["semantic_begin_id"])+ stop = int(self.manifest["im_end_id"])+ codebook_size = int(self.manifest["codebook_size"])+ assert logits is not None and hidden is not None+ for step in range(max_new_tokens):+ if stop_event is not None and stop_event.is_set():+ return+ semantic = self._sample_semantic(logits, previous, temperature, top_p, top_k, rng)+ if semantic == stop:+ return+ previous.append(semantic)+ previous = previous[-10:]++ fast_caches = self._empty_fast_caches()+ self._fast_step(hidden, 0, True, 0, fast_caches)+ first_code = semantic - begin+ if not 0 <= first_code < codebook_size:+ raise ValueError(f"semantic token {semantic} is outside the codebook range")+ codebooks = [first_code]+ token = first_code+ for fast_position in range(1, int(self.manifest["num_codebooks"])):+ fast_logits = self._fast_step(hidden, token, False, fast_position, fast_caches)+ token = _sample(fast_logits, temperature, top_p, top_k, rng)+ codebooks.append(token)+ frame = np.asarray(codebooks, dtype=np.int64)+ yield frame+ if step + 1 >= max_new_tokens:+ return+ column = np.concatenate([[semantic], frame]).reshape(1, -1, 1)+ logits, hidden = self._slow_step(column, prompt_len + step, slow_state)++ def decode_codes(self, codes: np.ndarray) -> np.ndarray:+ values = np.asarray(codes, dtype=np.int64)+ if values.ndim == 2:+ values = values[np.newaxis]+ expected = int(self.manifest["num_codebooks"])+ if values.ndim != 3 or values.shape[1] != expected or values.shape[2] == 0:+ raise ValueError(f"invalid generated codes shape: {values.shape}")+ decoder_input = self.decoder.get_inputs()[0]+ values = values.astype(_dtype(decoder_input.type), copy=False)+ audio = self.decoder.run(None, {decoder_input.name: values})[0]+ return np.asarray(audio, dtype=np.float32).reshape(-1)++ def synthesize(self, **kwargs: Any) -> tuple[np.ndarray, np.ndarray]:+ frames = list(self.iter_codes(**kwargs))+ if not frames:+ raise RuntimeError("model produced no codec frames")+ codes = np.stack(frames, axis=1)+ return self.decode_codes(codes), codes++ def stream(self, chunk_frames: int = 12, **kwargs: Any):+ all_frames: list[np.ndarray] = []+ emitted_samples = 0+ hop = int(self.manifest["codec_hop_length"])+ context = int(self.manifest.get("stream_context_frames", 128))+ guard = int(self.manifest.get("stream_guard_frames", 1)) * hop+ sequence = 0+ for frame in self.iter_codes(**kwargs):+ all_frames.append(frame)+ if len(all_frames) % max(1, int(chunk_frames)) != 0:+ continue+ start_frame = max(0, len(all_frames) - context - int(chunk_frames))+ window = np.stack(all_frames[start_frame:], axis=1)+ audio = self.decode_codes(window)+ absolute_start = start_frame * hop+ stable_end = absolute_start + max(0, audio.size - guard)+ begin = max(0, emitted_samples - absolute_start)+ end = max(begin, stable_end - absolute_start)+ if end > begin:+ chunk = np.ascontiguousarray(audio[begin:end])+ emitted_samples += chunk.size+ yield {+ "type": "audio_chunk",+ "seq": sequence,+ "audio": chunk,+ "frame_count": len(all_frames),+ }+ sequence += 1+ if not all_frames:+ raise RuntimeError("model produced no codec frames")+ start_frame = max(0, len(all_frames) - context - int(chunk_frames))+ audio = self.decode_codes(np.stack(all_frames[start_frame:], axis=1))+ absolute_start = start_frame * hop+ begin = max(0, emitted_samples - absolute_start)+ if begin < audio.size:+ yield {+ "type": "audio_chunk",+ "seq": sequence,+ "audio": np.ascontiguousarray(audio[begin:]),+ "frame_count": len(all_frames),+ }+ yield {"type": "complete", "codes": np.stack(all_frames, axis=1)}diff --git a/onnx_runtime_0_1b_int8/arktts_runtime/service.py b/onnx_runtime_0_1b_int8/arktts_runtime/service.pynew file mode 100644index 0000000..21a095f--- /dev/null+++ b/onnx_runtime_0_1b_int8/arktts_runtime/service.py@@ -0,0 +1,245 @@+from __future__ import annotations++import base64+import gc+import io+import json+import os+import threading+from pathlib import Path++import numpy as np+import soundfile as sf+from fastapi import FastAPI, File, Form, HTTPException, UploadFile+from fastapi.responses import Response, StreamingResponse+from pydantic import BaseModel, Field++from .registration import MAX_AUDIO_BYTES, VoiceRegistration+from .runtime import ArkTtsRuntime++ROOT = Path(__file__).resolve().parents[1]+MODEL_DIR = Path(os.getenv("ARKTTS_MODEL_DIR", str(ROOT / "model"))).expanduser()+VOICES_DIR = Path(os.getenv("ARKTTS_VOICES_DIR", str(ROOT / "voices"))).expanduser()+REGISTRATION_DIR = Path(+ os.getenv("ARKTTS_REGISTRATION_DIR", str(MODEL_DIR / "registration"))+).expanduser()+PRECISION = os.getenv("ARKTTS_PRECISION") or None+CODEC_PRECISION = os.getenv("ARKTTS_CODEC_PRECISION") or None+THREADS = int(os.getenv("ARKTTS_THREADS", "5"))++app = FastAPI(title="Audio8 0.1B INT8 ONNX Runtime")+runtime: ArkTtsRuntime | None = None+registration: VoiceRegistration | None = None+request_lock = threading.RLock()+active_stop: threading.Event | None = None+++class TtsRequest(BaseModel):+ text: str = Field(..., min_length=1, max_length=1000)+ voice_name: str = "default"+ max_new_tokens: int = Field(256, ge=1, le=2048)+ temperature: float = Field(0.7, gt=0.0, le=2.0)+ top_p: float = Field(0.9, gt=0.0, le=1.0)+ top_k: int = Field(50, ge=1, le=4096)+ seed: int = 42+++class OpenAiSpeechRequest(BaseModel):+ model: str = "arktts-0.1b"+ input: str = Field(..., min_length=1, max_length=1000)+ voice: str = "default"+ response_format: str = "wav"+++def _load_runtime() -> ArkTtsRuntime:+ global runtime, registration+ runtime = ArkTtsRuntime(MODEL_DIR, VOICES_DIR, PRECISION, CODEC_PRECISION, THREADS)+ registration = VoiceRegistration(+ REGISTRATION_DIR,+ runtime.voices.root,+ str(runtime.manifest["model_fingerprint"]),+ )+ return runtime++[email protected]_event("startup")+def startup() -> None:+ _load_runtime()+++def require_runtime() -> ArkTtsRuntime:+ if runtime is None:+ raise HTTPException(503, "runtime is not loaded")+ return runtime+++def _wav_response(audio: np.ndarray, sample_rate: int) -> Response:+ buffer = io.BytesIO()+ sf.write(buffer, np.asarray(audio, dtype=np.float32), int(sample_rate), format="WAV")+ return Response(buffer.getvalue(), media_type="audio/wav")++[email protected]("/api/health")+def health() -> dict:+ obj = require_runtime()+ return {+ "ok": True,+ "model": obj.manifest.get("model_id"),+ "precision": obj.precision,+ "codec_precision": obj.codec_precision,+ "providers": {+ "slow": obj.slow.get_providers(),+ "fast": obj.fast.get_providers(),+ "decoder": obj.decoder.get_providers(),+ },+ }++[email protected]("/api/voices")+def voices() -> dict:+ return {"voices": require_runtime().voices.list()}++[email protected]("/api/registration/status")+def registration_status() -> dict:+ if registration is None:+ raise HTTPException(503, "registration is not initialized")+ return registration.status()++[email protected]("/api/voices/register")+def register_voice(+ audio: UploadFile = File(...),+ text: str = Form(...),+ name: str = Form(...),+ overwrite: bool = Form(False),+) -> dict:+ global runtime+ if registration is None:+ raise HTTPException(503, "registration is not initialized")+ data = audio.file.read(MAX_AUDIO_BYTES + 1)+ with request_lock:+ current = require_runtime()+ precision, codec_precision = current.precision, current.codec_precision+ runtime = None+ del current+ gc.collect()+ try:+ meta = registration.register(+ data,+ audio.filename or "reference_audio",+ text,+ name,+ overwrite,+ )+ except FileExistsError as exc:+ raise HTTPException(409, str(exc)) from exc+ except ValueError as exc:+ raise HTTPException(400, str(exc)) from exc+ except RuntimeError as exc:+ raise HTTPException(503, str(exc)) from exc+ finally:+ runtime = ArkTtsRuntime(MODEL_DIR, VOICES_DIR, precision, codec_precision, THREADS)+ gc.collect()+ return {"ok": True, "voice": meta}++[email protected]("/api/tts")+def tts(request: TtsRequest) -> Response:+ with request_lock:+ obj = require_runtime()+ audio, _ = obj.synthesize(+ text=request.text,+ voice=request.voice_name,+ max_new_tokens=request.max_new_tokens,+ temperature=request.temperature,+ top_p=request.top_p,+ top_k=request.top_k,+ seed=request.seed,+ )+ return _wav_response(audio, int(obj.manifest["sample_rate"]))++[email protected]("/v1/audio/speech")+def openai_speech(request: OpenAiSpeechRequest) -> Response:+ if request.model not in {"arktts", "arktts-0.1b", "tts-1"}:+ raise HTTPException(400, f"unsupported model: {request.model}")+ if request.response_format not in {"wav", "pcm"}:+ raise HTTPException(400, "response_format must be wav or pcm")+ with request_lock:+ obj = require_runtime()+ audio, _ = obj.synthesize(text=request.input, voice=request.voice)+ if request.response_format == "pcm":+ pcm = (np.clip(audio, -1.0, 1.0) * 32767.0).astype("<i2").tobytes()+ return Response(pcm, media_type="application/octet-stream")+ return _wav_response(audio, int(obj.manifest["sample_rate"]))++[email protected]("/api/tts/stream")+def stream_tts(request: TtsRequest) -> StreamingResponse:+ def events():+ global active_stop+ stop = threading.Event()+ active_stop = stop+ try:+ with request_lock:+ obj = require_runtime()+ yield json.dumps(+ {+ "event": "start",+ "sample_rate": int(obj.manifest["sample_rate"]),+ "channels": 1,+ "sample_format": "s16le",+ }+ ) + "\n"+ for event in obj.stream(+ text=request.text,+ voice=request.voice_name,+ max_new_tokens=request.max_new_tokens,+ temperature=request.temperature,+ top_p=request.top_p,+ top_k=request.top_k,+ seed=request.seed,+ stop_event=stop,+ ):+ if stop.is_set():+ yield json.dumps({"event": "cancelled"}) + "\n"+ return+ if event["type"] == "audio_chunk":+ pcm = (np.clip(event["audio"], -1.0, 1.0) * 32767.0).astype("<i2")+ yield json.dumps(+ {+ "event": "audio_chunk",+ "seq": int(event["seq"]),+ "frame_count": int(event["frame_count"]),+ "pcm_b64": base64.b64encode(pcm.tobytes()).decode("ascii"),+ }+ ) + "\n"+ else:+ yield json.dumps(+ {"event": "complete", "frame_count": int(event["codes"].shape[1])}+ ) + "\n"+ finally:+ if active_stop is stop:+ active_stop = None++ return StreamingResponse(events(), media_type="application/x-ndjson")++[email protected]("/api/tts/cancel")+def cancel() -> dict[str, bool]:+ if active_stop is None:+ return {"ok": True, "cancelled": False}+ active_stop.set()+ return {"ok": True, "cancelled": True}++[email protected]("/api/runtime/reload")+def reload_runtime() -> dict:+ global runtime, registration+ with request_lock:+ runtime = None+ registration = None+ gc.collect()+ obj = _load_runtime()+ return {"ok": True, "precision": obj.precision, "codec_precision": obj.codec_precision}diff --git a/onnx_runtime_0_1b_int8/arktts_runtime/voices.py b/onnx_runtime_0_1b_int8/arktts_runtime/voices.pynew file mode 100644index 0000000..b087253--- /dev/null+++ b/onnx_runtime_0_1b_int8/arktts_runtime/voices.py@@ -0,0 +1,43 @@+from __future__ import annotations++import json+from pathlib import Path++import numpy as np+++class VoiceStore:+ def __init__(self, root: Path, num_codebooks: int, codebook_size: int = 4096):+ self.root = Path(root).resolve()+ self.num_codebooks = int(num_codebooks)+ self.codebook_size = int(codebook_size)++ def list(self) -> list[dict]:+ voices: list[dict] = []+ if not self.root.exists():+ return voices+ for meta_path in sorted(self.root.glob("*/meta.json")):+ try:+ voices.append(json.loads(meta_path.read_text(encoding="utf-8")))+ except (OSError, json.JSONDecodeError):+ continue+ return voices++ def load(self, name: str) -> tuple[np.ndarray, dict]:+ if not name or Path(name).name != name:+ raise ValueError("invalid voice name")+ voice_dir = self.root / name+ meta_path = voice_dir / "meta.json"+ codes_path = voice_dir / "codes.npy"+ if not meta_path.is_file() or not codes_path.is_file():+ raise KeyError(f"voice not found: {name}")+ meta = json.loads(meta_path.read_text(encoding="utf-8"))+ codes = np.load(codes_path, allow_pickle=False).astype(np.int64, copy=False)+ if codes.ndim != 2 or codes.shape[0] != self.num_codebooks or codes.shape[1] == 0:+ raise ValueError(f"invalid codes for voice {name}: {codes.shape}")+ if int(codes.min()) < 0 or int(codes.max()) >= self.codebook_size:+ raise ValueError(f"voice {name} contains code outside [0, {self.codebook_size})")+ reference_text = str(meta.get("reference_text", "")).strip()+ if not reference_text:+ raise ValueError(f"voice {name} has no reference_text")+ return codes, metadiff --git a/onnx_runtime_0_1b_int8/requirements.txt b/onnx_runtime_0_1b_int8/requirements.txtnew file mode 100644index 0000000..a2df4cd--- /dev/null+++ b/onnx_runtime_0_1b_int8/requirements.txt@@ -0,0 +1,10 @@+# Python 3.10+ is supported by this standalone 0.1B runtime.+numpy>=1.24,<3+onnxruntime>=1.22,<1.24+tokenizers>=0.20,<0.23+soundfile>=0.12,<1+scipy>=1.10,<2+fastapi>=0.110,<1+uvicorn>=0.29,<1+pydantic>=2.6,<3+python-multipart>=0.0.9,<1diff --git a/onnx_runtime_0_1b_int8/run_infer.sh b/onnx_runtime_0_1b_int8/run_infer.shnew file mode 100755index 0000000..f652811--- /dev/null+++ b/onnx_runtime_0_1b_int8/run_infer.sh@@ -0,0 +1,8 @@+#!/usr/bin/env bash+set -euo pipefail++ROOT="$(cd "$(dirname "$0")" && pwd)"+exec "$ROOT/.venv/bin/python" -m arktts_runtime.cli \+ --model-dir "${ARKTTS_MODEL_DIR:-$ROOT/model}" \+ --voices-dir "${ARKTTS_VOICES_DIR:-$ROOT/voices}" \+ "$@"diff --git a/onnx_runtime_0_1b_int8/run_server.sh b/onnx_runtime_0_1b_int8/run_server.shnew file mode 100755index 0000000..21ecde8--- /dev/null+++ b/onnx_runtime_0_1b_int8/run_server.sh@@ -0,0 +1,15 @@+#!/usr/bin/env bash+set -euo pipefail++ROOT="$(cd "$(dirname "$0")" && pwd)"+export ARKTTS_MODEL_DIR="${ARKTTS_MODEL_DIR:-$ROOT/model}"+export ARKTTS_VOICES_DIR="${ARKTTS_VOICES_DIR:-$ROOT/voices}"+export ARKTTS_REGISTRATION_DIR="${ARKTTS_REGISTRATION_DIR:-$ARKTTS_MODEL_DIR/registration}"+export ARKTTS_PRECISION="${ARKTTS_PRECISION:-int8}"+export ARKTTS_CODEC_PRECISION="${ARKTTS_CODEC_PRECISION:-fp16}"+export ARKTTS_THREADS="${ARKTTS_THREADS:-5}"++exec "$ROOT/.venv/bin/uvicorn" arktts_runtime.service:app \+ --app-dir "$ROOT" \+ --host "${HOST:-127.0.0.1}" \+ --port "${PORT:-8024}"diff --git a/onnx_runtime_0_1b_int8/scripts/register_default_voice.py b/onnx_runtime_0_1b_int8/scripts/register_default_voice.pynew file mode 100755index 0000000..2503f97--- /dev/null+++ b/onnx_runtime_0_1b_int8/scripts/register_default_voice.py@@ -0,0 +1,77 @@+#!/usr/bin/env python3+"""Create the bundled ``default`` voice from the model's reference codes."""++from __future__ import annotations++import argparse+import json+import os+import tempfile+from datetime import datetime, timezone+from pathlib import Path++import numpy as np+++def main() -> None:+ root = Path(__file__).resolve().parents[1]+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--model-dir", type=Path, default=root / "model")+ parser.add_argument("--voices-dir", type=Path, default=root / "voices")+ parser.add_argument("--name", default="default")+ parser.add_argument("--overwrite", action="store_true")+ args = parser.parse_args()++ manifest = json.loads((args.model_dir / "runtime_manifest.json").read_text(encoding="utf-8"))+ codes_path = args.model_dir / manifest.get("reference_codes", "reference_codes.npy")+ codes = np.load(codes_path, allow_pickle=False)+ expected = int(manifest["num_codebooks"])+ codebook_size = int(manifest.get("codebook_size", 4096))+ if (+ codes.ndim != 2+ or codes.shape[0] != expected+ or codes.shape[1] == 0+ or int(codes.min()) < 0+ or int(codes.max()) >= codebook_size+ ):+ raise ValueError(f"reference codes must have shape [{expected}, T] in [0, {codebook_size})")+ name = str(args.name).strip()+ if not name or Path(name).name != name or name in {".", ".."}:+ raise ValueError("name must be one path component")+ target = args.voices_dir / name+ if target.exists() and not args.overwrite:+ raise FileExistsError(f"voice already exists: {target}; pass --overwrite to replace it")++ args.voices_dir.mkdir(parents=True, exist_ok=True)+ temp = Path(tempfile.mkdtemp(prefix=f".{name}.", dir=args.voices_dir))+ try:+ np.save(temp / "codes.npy", codes.astype(np.uint16, copy=False))+ meta = {+ "name": name,+ "reference_text": str(manifest["reference_text"]),+ "shape": list(codes.shape),+ "dtype": "uint16",+ "sample_rate": int(manifest["sample_rate"]),+ "model_fingerprint": manifest["model_fingerprint"],+ "created_at": datetime.now(timezone.utc).isoformat(),+ "source_kind": "model_reference_codes",+ }+ (temp / "meta.json").write_text(+ json.dumps(meta, ensure_ascii=False, indent=2) + "\n",+ encoding="utf-8",+ )+ if target.exists():+ import shutil++ shutil.rmtree(target)+ os.replace(temp, target)+ finally:+ if temp.exists():+ import shutil++ shutil.rmtree(temp)+ print(f"registered {name}: shape={tuple(codes.shape)} at {target}")+++if __name__ == "__main__":+ main()diff --git a/onnx_runtime_0_1b_int8/setup.sh b/onnx_runtime_0_1b_int8/setup.shnew file mode 100755index 0000000..72cd5fb--- /dev/null+++ b/onnx_runtime_0_1b_int8/setup.sh@@ -0,0 +1,19 @@+#!/usr/bin/env bash+set -euo pipefail++ROOT="$(cd "$(dirname "$0")" && pwd)"+PYTHON_BIN="${PYTHON_BIN:-python3}"++if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then+ echo "Python was not found. Set PYTHON_BIN to a Python 3.10+ executable." >&2+ exit 1+fi+if ! "$PYTHON_BIN" -c 'import sys; raise SystemExit(sys.version_info < (3, 10))'; then+ echo "Python 3.10 or newer is required." >&2+ exit 1+fi++"$PYTHON_BIN" -m venv "$ROOT/.venv"+"$ROOT/.venv/bin/python" -m pip install --upgrade pip+"$ROOT/.venv/bin/python" -m pip install -r "$ROOT/requirements.txt"+echo "Environment ready: $ROOT/.venv"diff --git a/onnx_runtime_0_1b_int8/start_server.sh b/onnx_runtime_0_1b_int8/start_server.shnew file mode 100755index 0000000..2c03aa2--- /dev/null+++ b/onnx_runtime_0_1b_int8/start_server.sh@@ -0,0 +1,28 @@+#!/usr/bin/env bash+set -euo pipefail++ROOT="$(cd "$(dirname "$0")" && pwd)"+PID_FILE="$ROOT/.service.pid"+PORT="${PORT:-8024}"++if [[ -f "$PID_FILE" ]] && kill -0 "$(<"$PID_FILE")" 2>/dev/null; then+ echo "Service is already running with PID $(<"$PID_FILE")"+ exit 0+fi++PORT="$PORT" nohup "$ROOT/run_server.sh" >"$ROOT/service.log" 2>&1 &+PID=$!+echo "$PID" >"$PID_FILE"+for _ in {1..60}; do+ if curl -fsS "http://127.0.0.1:$PORT/api/health" >/dev/null 2>&1; then+ echo "Audio8 0.1B TTS is ready: http://127.0.0.1:$PORT"+ exit 0+ fi+ if ! kill -0 "$PID" 2>/dev/null; then+ echo "Service exited during startup. See $ROOT/service.log" >&2+ exit 1+ fi+ sleep 1+done+echo "Service did not become ready in 60 seconds. See $ROOT/service.log" >&2+exit 1diff --git a/onnx_runtime_0_1b_int8/stop_server.sh b/onnx_runtime_0_1b_int8/stop_server.shnew file mode 100755index 0000000..cbf1c4f--- /dev/null+++ b/onnx_runtime_0_1b_int8/stop_server.sh@@ -0,0 +1,17 @@+#!/usr/bin/env bash+set -euo pipefail++ROOT="$(cd "$(dirname "$0")" && pwd)"+PID_FILE="$ROOT/.service.pid"+if [[ ! -f "$PID_FILE" ]]; then+ echo "Service is not running"+ exit 0+fi+PID="$(<"$PID_FILE")"+if kill -0 "$PID" 2>/dev/null; then+ kill "$PID"+ echo "Stopped Audio8 0.1B TTS (PID $PID)"+else+ echo "Service PID $PID is not running"+fi+rm -f "$PID_FILE"diff --git a/onnx_runtime_0_1b_int8/tests/test_contract.py b/onnx_runtime_0_1b_int8/tests/test_contract.pynew file mode 100644index 0000000..80ac71b--- /dev/null+++ b/onnx_runtime_0_1b_int8/tests/test_contract.py@@ -0,0 +1,84 @@+from __future__ import annotations++import os+from pathlib import Path++import pytest++ort = pytest.importorskip("onnxruntime")++ROOT = Path(__file__).resolve().parents[1]+MODEL_DIR = Path(os.getenv("ARKTTS_MODEL_DIR", str(ROOT / "model")))+++def _session(name: str):+ path = MODEL_DIR / name+ if not path.is_file():+ pytest.skip(f"model file is not available: {path}")+ return ort.InferenceSession(str(path), providers=["CPUExecutionProvider"])+++def test_slow_hybrid_contract() -> None:+ session = _session("slow_ar_int8.onnx")+ inputs = {item.name: item for item in session.get_inputs()}+ assert set(inputs) == {+ "codes",+ "position",+ "cache_keys",+ "cache_values",+ "conv_states",+ "ssm_states",+ }+ assert tuple(inputs["codes"].shape) == (1, 11, 1)+ assert tuple(inputs["position"].shape) == (1,)+ assert tuple(inputs["cache_keys"].shape) == (24, 1, 2, 2048, 64)+ assert tuple(inputs["cache_values"].shape) == (24, 1, 2, 2048, 64)+ assert tuple(inputs["conv_states"].shape) == (24, 1, 896, 4)+ assert tuple(inputs["ssm_states"].shape) == (24, 1, 24, 32, 64)++ outputs = {item.name: item for item in session.get_outputs()}+ assert set(outputs) == {+ "logits",+ "hidden",+ "key_delta",+ "value_delta",+ "next_conv_states",+ "next_ssm_states",+ }+ assert tuple(outputs["logits"].shape) == (1, 1, 4097)+ assert tuple(outputs["hidden"].shape) == (1, 1, 512)+++def test_fast_and_decoder_contract() -> None:+ fast = _session("fast_ar_int8.onnx")+ fast_inputs = {item.name: item for item in fast.get_inputs()}+ assert {name for name in fast_inputs if name.startswith("cache_key_")} == {+ "cache_key_0",+ "cache_key_1",+ "cache_key_2",+ "cache_key_3",+ }+ assert {name for name in fast_inputs if name.startswith("cache_value_")} == {+ "cache_value_0",+ "cache_value_1",+ "cache_value_2",+ "cache_value_3",+ }+ assert tuple(fast_inputs["slow_hidden"].shape) == (1, 1, 512)+ assert tuple(fast_inputs["token_id"].shape) == (1, 1)+ assert tuple(fast_inputs["input_pos"].shape) == (1,)+ assert [item.name for item in fast.get_outputs()] == [+ "logits",+ "key_delta_0",+ "value_delta_0",+ "key_delta_1",+ "value_delta_1",+ "key_delta_2",+ "value_delta_2",+ "key_delta_3",+ "value_delta_3",+ ]+ assert tuple(fast.get_outputs()[0].shape) == (1, 1, 4096)++ decoder = _session("codec_decoder_fp16.onnx")+ assert tuple(decoder.get_inputs()[0].shape[1:2]) == (10,)diff --git a/onnx_runtime_0_1b_int8/tests/test_prompt.py b/onnx_runtime_0_1b_int8/tests/test_prompt.pynew file mode 100644index 0000000..c81bea5--- /dev/null+++ b/onnx_runtime_0_1b_int8/tests/test_prompt.py@@ -0,0 +1,27 @@+from __future__ import annotations++import numpy as np+import pytest++pytest.importorskip("tokenizers")++from arktts_runtime.prompt import clean_text, format_reference_text+++def test_text_normalization_preserves_special_tokens() -> None:+ assert clean_text("\x00 你好\n世界 ") == "你好世界"+ assert clean_text("你好\nworld") == "你好 world"+ assert format_reference_text("你好") == "<|speaker:0|>你好"+ assert format_reference_text("<|speaker:2|>hello") == "<|speaker:2|>hello"+++def test_sample_state_update_handles_full_and_delta_cache() -> None:+ from arktts_runtime.runtime import ArkTtsRuntime++ cache = np.zeros((1, 2, 10, 64), dtype=np.float32)+ delta = np.ones_like(cache[:, :, :1])+ ArkTtsRuntime._update_fast_cache(cache, delta, 3)+ assert np.all(cache[:, :, 3:4] == 1)+ replacement = np.full_like(cache, 2)+ ArkTtsRuntime._update_fast_cache(cache, replacement, 0)+ assert np.all(cache == 2)