High-Quality Voice Cloning TTS for 600+ Languages
Python
feat: add optional LoRA finetuning support via PEFT (#243)
Kawshik Kumar Paul38e992bc60f8
diff --git a/docs/lora_finetuning.md b/docs/lora_finetuning.mdnew file mode 100644index 0000000..1124f3f--- /dev/null+++ b/docs/lora_finetuning.md@@ -0,0 +1,194 @@+# LoRA Finetuning++In addition to full finetuning (see [training.md](training.md)), OmniVoice supports parameter-efficient finetuning via [LoRA](https://arxiv.org/abs/2106.09685) (Low-Rank Adaptation), using [PEFT](https://github.com/huggingface/peft). LoRA freezes the pretrained LLM backbone and trains small low-rank adapter matrices instead — typically **3-5% of parameters trainable** — which means:++- Much lower optimizer memory (Adam keeps 2 extra tensors per trainable parameter).+- Much smaller adapter checkpoints to ship (tens of MB vs. the full model size).+- The original pretrained weights stay untouched on disk; the adapter can be dropped to recover the original model exactly.++LoRA is **opt-in** via config — with `use_lora: false` (the default), training behaves exactly as documented in [training.md](training.md).++## Installation++```bash+pip install "omnivoice[lora]"+# or, from a source checkout:+pip install peft+```++## What Gets Adapted++| Config field | Applies to | Why |+|---|---|---|+| `lora_target_modules` | LLM attention/MLP projections (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` by default) | This is where the pretrained language/text-conditioning knowledge lives — the standard set of layers LoRA adapts for transformer LLMs. |+| `lora_modules_to_save` | `audio_embeddings`, `audio_heads` by default | These are OmniVoice-specific audio token I/O layers, not part of the pretrained LLM. There's no pretrained knowledge in them to preserve via low-rank adapters, so they're kept **fully trainable** instead and saved alongside the adapter. |++Everything else (the LLM's other weights, layernorms, etc.) stays frozen, exactly as loaded from the base checkpoint.++## Configuration++Start from [examples/config/train_config_finetune_lora.json](../examples/config/train_config_finetune_lora.json) — identical to [train_config_finetune_sdpa.json](../examples/config/train_config_finetune_sdpa.json) plus the LoRA fields:++```json+{+ "init_from_checkpoint": "k2-fsa/OmniVoice",++ "use_lora": true,+ "lora_r": 16,+ "lora_alpha": 32,+ "lora_dropout": 0.05,+ "lora_bias": "none",+ "lora_target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],+ "lora_modules_to_save": ["audio_embeddings", "audio_heads"],++ "learning_rate": 1e-4,+ "attn_implementation": "sdpa"+}+```++| Field | Description | Default |+|---|---|---|+| `use_lora` | Enable LoRA finetuning instead of full finetuning | `false` |+| `lora_r` | LoRA rank — higher = more capacity, more trainable params | `16` |+| `lora_alpha` | LoRA scaling factor (effective scale is `lora_alpha / lora_r`) | `32` |+| `lora_dropout` | Dropout applied inside the LoRA adapter path | `0.05` |+| `lora_bias` | Which biases to train: `"none"`, `"all"`, or `"lora_only"` | `"none"` |+| `lora_target_modules` | LLM submodule names to attach LoRA adapters to | attention + MLP projections |+| `lora_modules_to_save` | Non-LLM modules trained in full instead of via LoRA | `audio_embeddings`, `audio_heads` |++A higher `learning_rate` than full finetuning (`1e-4` vs. `5e-5`) is typically used since the adapter matrices are randomly initialized and trained from scratch, rather than finetuning already-converged pretrained weights — a higher LR converges faster without the instability risk full finetuning has at that LR.++## Launching Training++```bash+accelerate launch \+ --gpu_ids "0,1" \+ --num_processes 2 \+ -m omnivoice.cli.train \+ --train_config examples/config/train_config_finetune_lora.json \+ --data_config examples/config/data_config_finetune.json \+ --output_dir exp/omnivoice_finetune_lora+```++Or run the full example pipeline (tokenization + training):++```bash+bash examples/run_finetune_lora.sh+```++On startup you'll see a line like:++```text+trainable params: 19,316,736 || all params: 631,894,016 || trainable%: 3.0570+```++confirming only the adapters plus `audio_embeddings`/`audio_heads` are being trained.++## How Many Checkpoints Get Saved++Same three fields as full finetuning govern this:++| Field | Effect |+|---|---|+| `steps` | Total training steps |+| `save_steps` | A checkpoint is saved every `save_steps` steps |+| `keep_last_n_checkpoints` | `-1` keeps every checkpoint; a positive `N` rotates old ones out, keeping only the most recent `N` |++Number of checkpoints saved = `steps / save_steps` (when `steps` is a multiple of `save_steps`), capped at `keep_last_n_checkpoints` if positive.++## Resuming Training++Identical mechanism to full finetuning — set `resume_from_checkpoint`:++```json+{+ "resume_from_checkpoint": "exp/omnivoice_finetune_lora/checkpoint-500"+}+```++Each run rebuilds the same LoRA-wrapped model architecture from `init_from_checkpoint` plus the `lora_*` config fields, then restores optimizer/scheduler/RNG state and adapter weights from the checkpoint on top of it. Keep `init_from_checkpoint` and the `lora_*` fields unchanged across a resume — a mismatch will surface as a shape/key error when `accelerate` restores state.++## Adapter Inference++Apply a trained adapter on top of the base model at inference time — merged into the model in-memory (not written to disk), so generation uses the regular `OmniVoice.generate()` path unmodified:++```bash+omnivoice-infer \+ --model k2-fsa/OmniVoice \+ --lora_adapter exp/omnivoice_finetune_lora/checkpoint-500 \+ --text "Hello, this is a text for text-to-speech." \+ --ref_audio ref.wav --ref_text "Reference transcript." \+ --output out.wav+```++`--lora_adapter` accepts any directory containing `adapter_config.json` + `adapter_model.safetensors` — i.e. any LoRA training checkpoint directory.++## Adapter Merging++To deploy with **no PEFT dependency at inference time**, merge the adapter into the base model once and save the result as a normal, standalone OmniVoice checkpoint:++```bash+omnivoice-merge-lora \+ --base_model k2-fsa/OmniVoice \+ --lora_adapter exp/omnivoice_finetune_lora/checkpoint-500 \+ --output_dir exp/omnivoice_finetune_lora/merged+```++This loads the base model, merges the LoRA deltas into its weights, and writes a complete, self-contained OmniVoice directory (model weights, tokenizer, and audio tokenizer).++## Using the Merged Model++The merged directory is a normal OmniVoice checkpoint — use it exactly like any other:++```bash+omnivoice-infer \+ --model exp/omnivoice_finetune_lora/merged \+ --text "Hello, this is a text for text-to-speech." \+ --instruct "male, British accent" \+ --output out.wav+```++```python+from omnivoice.models.omnivoice import OmniVoice++model = OmniVoice.from_pretrained("exp/omnivoice_finetune_lora/merged", device_map="cuda:0")+audios = model.generate(text="Hello world", language="en")+```++The merged model is saved in whatever dtype the base model was loaded in when merging (`float32` by default, since `omnivoice-merge-lora` loads the base model with `dtype=torch.float32`). Pass `--dtype fp32` / `dtype=torch.float32` at inference to match exactly, or `fp16`/`bf16` for less VRAM and faster inference — `from_pretrained` casts down automatically, and quality differences are usually negligible.++## Checkpoint Structure++A LoRA training checkpoint (`exp/.../checkpoint-<step>/`) contains two independent sets of files:++```text+checkpoint-500/+├── adapter_config.json # PEFT LoRA config (small)+├── adapter_model.safetensors # LoRA weights + audio_embeddings/audio_heads (small)+├── tokenizer.json # text tokenizer, saved for adapter/merge use+├── tokenizer_config.json+├── model.safetensors # accelerate's full training state (base + adapter weights)+├── optimizer.bin # optimizer state, for resuming+├── scheduler.bin # LR scheduler state, for resuming+└── random_states_0.pkl # RNG state, for resuming+```++- **For adapter inference / merging**, only `adapter_config.json` + `adapter_model.safetensors` (+ tokenizer) are used.+- **For resuming training**, `accelerate`'s own state files (`model.safetensors`, `optimizer.bin`, `scheduler.bin`, `random_states_*.pkl`) are used — these include the full (frozen + trainable) model state, since `accelerate`'s checkpointing does not distinguish frozen from trainable parameters. This makes `checkpoint-*/` directories larger than the adapter alone, but keeps resuming exactly as reliable as full finetuning's. If you only need the adapter for deployment, you can delete everything except `adapter_config.json`/`adapter_model.safetensors`/the tokenizer files once training is done.++## Troubleshooting++**`ModuleNotFoundError: No module named 'peft'`**+Run `pip install peft` (or `pip install "omnivoice[lora]"`).++**Resume fails with a shape/key mismatch error**+`resume_from_checkpoint` must be resumed with the *same* `init_from_checkpoint`, `lora_r`, `lora_alpha`, `lora_target_modules`, and `lora_modules_to_save` as the run that produced the checkpoint — these determine the model architecture `accelerate`'s state restore expects to find.++**`omnivoice-merge-lora` / `--lora_adapter` can't find the adapter**+Point it at a checkpoint *directory*, not a specific file — it must contain `adapter_config.json`.++**Training loss looks the same as a frozen model / doesn't move**+Check the `trainable params: ...` line printed at startup. If it's `0` or doesn't include LoRA layers, verify `use_lora: true` is set in the config passed to `--train_config`, and that `lora_target_modules` names match submodules in your `llm_name_or_path`'s architecture.++**Checkpoint saves or merging are unexpectedly slow**+Each checkpoint writes a full base-model-sized file (accelerate's training state — see [Checkpoint Structure](#checkpoint-structure)), and merging both reads and writes one. If disk space is tight or I/O is contended, these can take much longer than expected; check available disk space first.diff --git a/docs/training.md b/docs/training.mdindex 9339e51..d89dfc9 100644--- a/docs/training.md+++ b/docs/training.md@@ -94,6 +94,10 @@ To start training from a pretrained OmniVoice checkpoint (for fine-tuning):}```+## LoRA Finetuning++For parameter-efficient finetuning, set `"use_lora": true` in your training config to train low-rank adapters instead of the full model. See [lora_finetuning.md](lora_finetuning.md) for configuration, adapter inference, and merging back into a standalone checkpoint.+## MonitoringTraining logs to TensorBoard:diff --git a/examples/README.md b/examples/README.mdindex 00f6485..566a5ec 100644--- a/examples/README.md+++ b/examples/README.md@@ -6,6 +6,7 @@ This directory contains scripts and configs for training, fine-tuning, and evalu|---|---|---|| Training from scratch | [run_emilia.sh](run_emilia.sh) | Full pipeline on the Emilia dataset (data check, tokenization, training) || Fine-tuning | [run_finetune.sh](run_finetune.sh) | Fine-tune from a pretrained checkpoint using your own JSONL data |+| LoRA fine-tuning | [run_finetune_lora.sh](run_finetune_lora.sh) | Parameter-efficient finetuning via PEFT LoRA adapters — see [../docs/lora_finetuning.md](../docs/lora_finetuning.md) || Evaluation | [run_eval.sh](run_eval.sh) | Evaluate WER, speaker similarity, and UTMOS on standard test sets |---diff --git a/examples/config/train_config_finetune_lora.json b/examples/config/train_config_finetune_lora.jsonnew file mode 100644index 0000000..3c02600--- /dev/null+++ b/examples/config/train_config_finetune_lora.json@@ -0,0 +1,52 @@+{+ "llm_name_or_path": "Qwen/Qwen3-0.6B",+ "audio_vocab_size": 1025,+ "audio_mask_id": 1024,+ "num_audio_codebook": 8,++ "audio_codebook_weights": [8, 8, 6, 6, 4, 4, 2, 2],+ "drop_cond_ratio": 0.1,+ "prompt_ratio_range": [0.0, 0.3],+ "mask_ratio_range": [0.0, 1.0],+ "language_ratio": 0.8,+ "use_pinyin_ratio": 0.0,+ "instruct_ratio": 0.0,+ "only_instruct_ratio": 0.0,++ "resume_from_checkpoint": null,+ "init_from_checkpoint": "k2-fsa/OmniVoice",++ "use_lora": true,+ "lora_r": 16,+ "lora_alpha": 32,+ "lora_dropout": 0.05,+ "lora_bias": "none",+ "lora_target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],+ "lora_modules_to_save": ["audio_embeddings", "audio_heads"],++ "learning_rate": 1e-4,+ "weight_decay": 0.01,+ "max_grad_norm": 1.0,+ "steps": 5000,+ "seed": 42,+ "warmup_type": "ratio",+ "warmup_ratio": 0.01,+ "warmup_steps": 0,++ "attn_implementation": "sdpa",+ "max_sample_tokens": 2000,+ "min_sample_tokens": 50,+ "max_batch_size": 64,++ "batch_tokens": 8192,+ "gradient_accumulation_steps": 1,+ "num_workers": 2,++ "mixed_precision": "bf16",+ "allow_tf32": true,++ "logging_steps": 50,+ "eval_steps": 500,+ "save_steps": 500,+ "keep_last_n_checkpoints": -1+}diff --git a/examples/run_finetune_lora.sh b/examples/run_finetune_lora.shnew file mode 100755index 0000000..5bf200a--- /dev/null+++ b/examples/run_finetune_lora.sh@@ -0,0 +1,85 @@+#!/bin/bash++# This script demonstrates how to LoRA fine-tune OmniVoice from a JSONL manifest.+# See ../docs/lora_finetuning.md for the full guide (why each step is required,+# adapter inference, and adapter merging for deployment).++set -euo pipefail++stage=0+stop_stage=1++# ====== Modify as needed ======+# GPUs to use+GPU_IDS="0,1"+NUM_GPUS=2++# Path to your input JSONL file+# (each line: {"id": ..., "audio_path": ..., "text": ..., "language_id": ...})+TRAIN_JSONL="data/my_data_train.jsonl"++# Path to your dev JSONL file. Set to empty string to skip dev set.+DEV_JSONL="data/my_data_dev.jsonl"++# Directory to write tokenized WebDataset shards+TOKEN_DIR="data/finetune/tokens"++# Audio tokenizer model (HuggingFace repo or local path)+TOKENIZER_PATH="eustlb/higgs-audio-v2-tokenizer"++# LoRA training config file+TRAIN_CONFIG="config/train_config_finetune_lora.json"++# Data config file+data_config="config/data_config_finetune.json"++# Output directory for the LoRA adapter checkpoints+OUTPUT_DIR="exp/omnivoice_finetune_lora"+# =================================++export PYTHONPATH="$(cd "$(dirname "$0")/.." && pwd):${PYTHONPATH:-}"+++# Stage 0: Tokenize audio into WebDataset shards+if [ $stage -le 0 ] && [ $stop_stage -ge 0 ]; then+ echo "Stage 0: Tokenizing audio"++ for split_jsonl_path in ${TRAIN_JSONL} ${DEV_JSONL}; do+ if [ -z "${split_jsonl_path}" ]; then+ continue+ fi++ if [ "${split_jsonl_path}" = "${TRAIN_JSONL}" ]; then+ split="train"+ else+ split="dev"+ fi++ echo " Tokenizing ${split} from ${split_jsonl_path}"++ CUDA_VISIBLE_DEVICES=${GPU_IDS} \+ python -m omnivoice.scripts.extract_audio_tokens \+ --input_jsonl "${split_jsonl_path}" \+ --tar_output_pattern "${TOKEN_DIR}/${split}/audios/shard-%06d.tar" \+ --jsonl_output_pattern "${TOKEN_DIR}/${split}/txts/shard-%06d.jsonl" \+ --tokenizer_path "${TOKENIZER_PATH}" \+ --nj_per_gpu 3 \+ --shuffle True++ echo " Done. Manifest written to ${TOKEN_DIR}/${split}/data.lst"+ done+fi+++# Stage 1: LoRA fine-tune+if [ $stage -le 1 ] && [ $stop_stage -ge 1 ]; then+ echo "Stage 1: LoRA fine-tuning"++ accelerate launch \+ --gpu_ids "${GPU_IDS}" \+ --num_processes ${NUM_GPUS} \+ -m omnivoice.cli.train \+ --train_config ${TRAIN_CONFIG} \+ --data_config ${data_config} \+ --output_dir ${OUTPUT_DIR}+fidiff --git a/omnivoice/cli/infer.py b/omnivoice/cli/infer.pyindex ce4657d..2d903cc 100644--- a/omnivoice/cli/infer.py+++ b/omnivoice/cli/infer.py@@ -53,6 +53,14 @@ def get_parser() -> argparse.ArgumentParser:required=True,help="Output WAV file path.",)+ parser.add_argument(+ "--lora_adapter",+ type=str,+ default=None,+ help="Path to a LoRA adapter directory (e.g. a LoRA training "+ "checkpoint) to apply on top of --model. Merged in-memory before "+ "generation.",+ )# Voice cloningparser.add_argument("--ref_audio",@@ -122,6 +130,12 @@ def main():args.model, device_map=device, dtype=torch.float16)+ if args.lora_adapter:+ from omnivoice.utils.lora import load_lora_adapter++ logging.info(f"Applying LoRA adapter from {args.lora_adapter} ...")+ model = load_lora_adapter(model, args.lora_adapter)+logging.info(f"Generating audio for: {args.text[:80]}...")audios = model.generate(text=args.text,diff --git a/omnivoice/cli/merge_lora.py b/omnivoice/cli/merge_lora.pynew file mode 100644index 0000000..6e5d86f--- /dev/null+++ b/omnivoice/cli/merge_lora.py@@ -0,0 +1,123 @@+#!/usr/bin/env python3+# Copyright 2026 Xiaomi Corp. (authors: Han Zhu)+#+# See ../../LICENSE for clarification regarding multiple authors+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Merge a LoRA adapter into its base model for normal OmniVoice deployment.++Loads the base model, merges the LoRA adapter weights into it, and saves the+result as a plain HuggingFace-format OmniVoice checkpoint (config.json ++model.safetensors + tokenizer + audio_tokenizer) that can be loaded directly+with ``OmniVoice.from_pretrained()`` — no PEFT dependency needed at inference+time.++Usage:+ omnivoice-merge-lora \\+ --base_model /path/to/OmniVoice \\+ --lora_adapter exp/omnivoice_lora/checkpoint-5000 \\+ --output_dir exp/omnivoice_lora/merged+"""++import argparse+import logging+import os+import shutil++import torch+from transformers import AutoTokenizer++from omnivoice.models.omnivoice import OmniVoice, _resolve_model_path+from omnivoice.utils.lora import load_lora_adapter++logger = logging.getLogger(__name__)++# Files that are part of a deployable OmniVoice directory but are not part of+# the model's state dict, so save_pretrained() does not (re)write them.+_EXTRA_DEPLOY_PATHS = ["audio_tokenizer", "chat_template.jinja"]+++def get_parser() -> argparse.ArgumentParser:+ parser = argparse.ArgumentParser(+ description="Merge a LoRA adapter into its base OmniVoice model",+ formatter_class=argparse.ArgumentDefaultsHelpFormatter,+ )+ parser.add_argument(+ "--base_model",+ type=str,+ required=True,+ help="Base model checkpoint path (local dir or HuggingFace repo id) "+ "the LoRA adapter was trained from (i.e. its init_from_checkpoint).",+ )+ parser.add_argument(+ "--lora_adapter",+ type=str,+ required=True,+ help="Path to the LoRA adapter directory (e.g. a LoRA training "+ "checkpoint) containing adapter_config.json.",+ )+ parser.add_argument(+ "--output_dir",+ type=str,+ required=True,+ help="Directory to write the merged, deployable OmniVoice model to.",+ )+ return parser+++def main():+ logging.basicConfig(+ format="%(asctime)s %(levelname)s [%(filename)s:%(lineno)d] %(message)s",+ level=logging.INFO,+ )+ args = get_parser().parse_args()++ logger.info(f"Loading base model from {args.base_model} ...")+ model = OmniVoice.from_pretrained(args.base_model, train=True, dtype=torch.float32)++ merged = load_lora_adapter(model, args.lora_adapter, merge=True)++ os.makedirs(args.output_dir, exist_ok=True)+ logger.info(f"Saving merged model to {args.output_dir} ...")+ merged.save_pretrained(args.output_dir)++ # Prefer the tokenizer saved alongside the adapter (training checkpoints+ # save it there); it is identical to the base model's unless new special+ # tokens were added, in which case it is the correct one to keep.+ tokenizer_source = (+ args.lora_adapter+ if os.path.exists(os.path.join(args.lora_adapter, "tokenizer_config.json"))+ else args.base_model+ )+ tokenizer = AutoTokenizer.from_pretrained(tokenizer_source)+ tokenizer.save_pretrained(args.output_dir)++ # Copy over deployment files that aren't part of the model state dict+ # (audio tokenizer, chat template) so the output directory is a+ # self-contained, directly loadable OmniVoice checkpoint.+ resolved_base = _resolve_model_path(args.base_model)+ for name in _EXTRA_DEPLOY_PATHS:+ src = os.path.join(resolved_base, name)+ dst = os.path.join(args.output_dir, name)+ if os.path.exists(src) and not os.path.exists(dst):+ if os.path.isdir(src):+ shutil.copytree(src, dst)+ else:+ shutil.copy2(src, dst)++ logger.info(f"Merged model saved to {args.output_dir}")+++if __name__ == "__main__":+ main()diff --git a/omnivoice/training/builder.py b/omnivoice/training/builder.pyindex 6a33431..72c50ff 100644--- a/omnivoice/training/builder.py+++ b/omnivoice/training/builder.py@@ -33,6 +33,7 @@ Key functions:"""import logging+import osfrom functools import partialfrom typing import Tuple@@ -125,9 +126,37 @@ def build_model_and_tokenizer(model.config.bos_token_id = tokenizer.bos_token_idmodel.config.eos_token_id = tokenizer.eos_token_id+ # 5. Wrap with LoRA adapters (optional)+ if config.use_lora:+ model = _apply_lora(model, config)+return model, tokenizer+def _apply_lora(model: OmniVoice, config: TrainingConfig) -> OmniVoice:+ """Wrap the model with PEFT LoRA adapters.++ ``lora_target_modules`` (LLM attention/MLP projections) are trained as+ low-rank adapters; ``lora_modules_to_save`` (the audio embedding and+ output head) are OmniVoice-specific I/O layers with no LLM-pretrained+ knowledge to preserve, so they are kept fully trainable instead.+ """+ from peft import LoraConfig, get_peft_model++ lora_config = LoraConfig(+ r=config.lora_r,+ lora_alpha=config.lora_alpha,+ lora_dropout=config.lora_dropout,+ bias=config.lora_bias,+ target_modules=config.lora_target_modules,+ modules_to_save=config.lora_modules_to_save,+ )+ model = get_peft_model(model, lora_config)+ if int(os.environ.get("LOCAL_RANK", "0")) == 0:+ model.print_trainable_parameters()+ return model++def build_dataloaders(config: TrainingConfig, tokenizer: AutoTokenizer) -> Tuple[DataLoader, DataLoader]:diff --git a/omnivoice/training/config.py b/omnivoice/training/config.pyindex 49cd880..9756451 100644--- a/omnivoice/training/config.py+++ b/omnivoice/training/config.py@@ -55,6 +55,30 @@ class TrainingConfig:resume_from_checkpoint: Optional[str] = Noneinit_from_checkpoint: Optional[str] = None+ # LoRA (PEFT) Settings+ use_lora: bool = False+ lora_r: int = 16+ lora_alpha: int = 32+ lora_dropout: float = 0.05+ lora_bias: str = "none"+ lora_target_modules: List[str] = field(+ default_factory=lambda: [+ "q_proj",+ "k_proj",+ "v_proj",+ "o_proj",+ "gate_proj",+ "up_proj",+ "down_proj",+ ]+ )+ # Non-LLM modules trained in full rather than via low-rank adapters,+ # since they are OmniVoice-specific audio I/O layers, not general LLM+ # knowledge, and are small relative to the LLM backbone.+ lora_modules_to_save: List[str] = field(+ default_factory=lambda: ["audio_embeddings", "audio_heads"]+ )+# Training Hyperparamslearning_rate: float = 1e-4weight_decay: float = 0.01diff --git a/omnivoice/training/trainer.py b/omnivoice/training/trainer.pyindex ec015be..9ea53e6 100644--- a/omnivoice/training/trainer.py+++ b/omnivoice/training/trainer.py@@ -164,8 +164,11 @@ class OmniTrainer:def create_optimizer_and_scheduler(self):"""Default AdamW + configurable LR Scheduler."""+ # Only trainable parameters (all of them for full fine-tuning; LoRA+ # adapters + modules_to_save when config.use_lora froze the rest).+ trainable_params = [p for p in self.model.parameters() if p.requires_grad]optimizer = torch.optim.AdamW(- self.model.parameters(),+ trainable_params,lr=self.config.learning_rate,weight_decay=self.config.weight_decay,)diff --git a/omnivoice/utils/lora.py b/omnivoice/utils/lora.pynew file mode 100644index 0000000..905128e--- /dev/null+++ b/omnivoice/utils/lora.py@@ -0,0 +1,59 @@+#!/usr/bin/env python3+# Copyright 2026 Xiaomi Corp. (authors: Han Zhu)+#+# See ../../LICENSE for clarification regarding multiple authors+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""LoRA adapter loading shared by inference and merge CLIs.++Used by ``omnivoice.cli.infer`` (``--lora_adapter``) and+``omnivoice.cli.merge_lora``.+"""++import logging+from typing import Union++from omnivoice.models.omnivoice import OmniVoice++logger = logging.getLogger(__name__)+++def load_lora_adapter(+ model: OmniVoice, adapter_path: str, merge: bool = True+) -> Union[OmniVoice, "peft.PeftModel"]: # noqa: F821+ """Attach a trained LoRA adapter to a loaded ``OmniVoice`` model.++ Args:+ model: An ``OmniVoice`` model, as returned by+ :meth:`OmniVoice.from_pretrained`.+ adapter_path: Path to a LoRA adapter directory containing+ ``adapter_config.json`` and ``adapter_model.safetensors``, e.g.+ a LoRA training checkpoint directory.+ merge: If ``True`` (default), merge the adapter weights into+ ``model`` in-memory and return the plain ``OmniVoice`` model, so+ its custom methods (``generate``, ``create_voice_clone_prompt``,+ etc.) keep working unmodified. If ``False``, return the+ ``peft.PeftModel`` wrapper with the adapter kept separate.++ Returns:+ The merged ``OmniVoice`` model (``merge=True``) or the wrapping+ ``PeftModel`` (``merge=False``).+ """+ from peft import PeftModel++ logger.info("Loading LoRA adapter from %s", adapter_path)+ peft_model = PeftModel.from_pretrained(model, adapter_path)+ if merge:+ return peft_model.merge_and_unload()+ return peft_modeldiff --git a/pyproject.toml b/pyproject.tomlindex bd7f04f..b93124b 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -59,10 +59,15 @@ eval = ["unidecode", # Unicode normalization]+lora = [+ "peft>=0.20.0", # LoRA fine-tuning support+]+[project.scripts]omnivoice-infer = "omnivoice.cli.infer:main"omnivoice-infer-batch = "omnivoice.cli.infer_batch:main"omnivoice-demo = "omnivoice.cli.demo:main"+omnivoice-merge-lora = "omnivoice.cli.merge_lora:main"[project.urls]Homepage = "https://github.com/k2-fsa/OmniVoice"diff --git a/tests/test_lora.py b/tests/test_lora.pynew file mode 100644index 0000000..acb6762--- /dev/null+++ b/tests/test_lora.py@@ -0,0 +1,225 @@+#!/usr/bin/env python3+# Copyright 2026 Xiaomi Corp. (authors: Han Zhu)+#+# See ../LICENSE for clarification regarding multiple authors+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Tests for LoRA fine-tuning support (config, model wrapping, checkpoint+save/resume, and adapter merging).++Requires a local OmniVoice model directory, set via the+``OMNIVOICE_TEST_MODEL_PATH`` environment variable. Tests are skipped if it+is not set or does not exist, since they load the actual pretrained model.+"""++import json+import os++import pytest+import torch++MODEL_PATH = os.environ.get("OMNIVOICE_TEST_MODEL_PATH")++pytestmark = pytest.mark.skipif(+ not MODEL_PATH or not os.path.isdir(MODEL_PATH),+ reason="Set OMNIVOICE_TEST_MODEL_PATH to a local OmniVoice model directory to run these tests.",+)+++def _lora_config(output_dir, **overrides):+ from omnivoice.training.config import TrainingConfig++ kwargs = {+ "llm_name_or_path": MODEL_PATH,+ "init_from_checkpoint": MODEL_PATH,+ "attn_implementation": "sdpa",+ "use_lora": True,+ "lora_r": 4,+ "lora_alpha": 8,+ "output_dir": output_dir,+ "steps": 2,+ }+ kwargs.update(overrides)+ return TrainingConfig(**kwargs)+++def test_training_config_lora_defaults():+ from omnivoice.training.config import TrainingConfig++ config = TrainingConfig()+ assert config.use_lora is False+ assert config.lora_r == 16+ assert "q_proj" in config.lora_target_modules+ assert "audio_embeddings" in config.lora_modules_to_save+ assert "audio_heads" in config.lora_modules_to_save+++def test_training_config_lora_json_roundtrip(tmp_path):+ from omnivoice.training.config import TrainingConfig++ config = _lora_config(str(tmp_path))+ json_path = tmp_path / "train_config.json"+ config.save_to_json(str(json_path))++ with open(json_path) as f:+ data = json.load(f)+ assert data["use_lora"] is True+ assert data["lora_r"] == 4++ reloaded = TrainingConfig.from_json(str(json_path))+ assert reloaded.use_lora is True+ assert reloaded.lora_r == 4+++def test_build_model_with_lora_freezes_base_and_keeps_io_heads_trainable(tmp_path):+ from omnivoice.training.builder import build_model_and_tokenizer++ config = _lora_config(str(tmp_path))+ model, _tokenizer = build_model_and_tokenizer(config)++ from peft import PeftModel++ assert isinstance(model, PeftModel)++ trainable = {n for n, p in model.named_parameters() if p.requires_grad}+ assert any("lora_A" in n for n in trainable)+ assert any("lora_B" in n for n in trainable)+ # audio_embeddings/audio_heads are modules_to_save -> fully trainable+ assert any("audio_embeddings" in n for n in trainable)+ assert any("audio_heads" in n for n in trainable)+ # base LLM linear weights themselves (not the lora_A/B deltas) stay frozen+ frozen = {n for n, p in model.named_parameters() if not p.requires_grad}+ assert any(n.endswith("self_attn.q_proj.base_layer.weight") for n in frozen)+++def test_build_model_without_lora_is_plain_omnivoice(tmp_path):+ from omnivoice.models.omnivoice import OmniVoice+ from omnivoice.training.builder import build_model_and_tokenizer++ config = _lora_config(str(tmp_path), use_lora=False)+ model, _tokenizer = build_model_and_tokenizer(config)++ assert type(model) is OmniVoice+ assert all(p.requires_grad for p in model.parameters())+++def test_lora_forward_and_backward(tmp_path):+ from omnivoice.training.builder import build_model_and_tokenizer++ config = _lora_config(str(tmp_path))+ model, _tokenizer = build_model_and_tokenizer(config)++ batch, num_codebooks, seq_len, cond_len = 1, 8, 20, 10+ input_ids = torch.randint(0, 100, (batch, num_codebooks, seq_len))+ audio_mask = torch.zeros(batch, seq_len, dtype=torch.bool)+ audio_mask[:, cond_len:] = True+ labels = torch.full((batch, num_codebooks, seq_len), -100, dtype=torch.long)+ labels[:, :, cond_len:] = torch.randint(+ 0, 100, (batch, num_codebooks, seq_len - cond_len)+ )++ out = model(input_ids=input_ids, audio_mask=audio_mask, labels=labels)+ assert out.loss is not None and torch.isfinite(out.loss)+ out.loss.backward()++ grad_params = [p for p in model.parameters() if p.grad is not None]+ assert len(grad_params) > 0+++def test_lora_checkpoint_save_and_resume(tmp_path):+ from accelerate import Accelerator++ from omnivoice.training.builder import build_model_and_tokenizer+ from omnivoice.training.checkpoint import load_checkpoint, save_checkpoint++ output_dir = str(tmp_path / "exp")+ config = _lora_config(output_dir)++ model, tokenizer = build_model_and_tokenizer(config)+ accelerator = Accelerator(project_dir=output_dir)+ os.makedirs(output_dir, exist_ok=True)++ optimizer = torch.optim.AdamW(+ [p for p in model.parameters() if p.requires_grad], lr=1e-4+ )+ model, optimizer = accelerator.prepare(model, optimizer)++ save_checkpoint(accelerator, model, tokenizer, output_dir, step=1, keep_last_n=-1)++ checkpoint_dir = os.path.join(output_dir, "checkpoint-1")+ saved_files = set(os.listdir(checkpoint_dir))+ # Small PEFT adapter files, used for inference/merging.+ assert {"adapter_config.json", "adapter_model.safetensors"} <= saved_files+ # Full accelerator training state, used for resuming.+ assert {"optimizer.bin"} <= saved_files++ # Resuming requires rebuilding the same LoRA architecture first, then+ # restoring accelerator state on top of it (mirrors full fine-tuning).+ model2, _tokenizer2 = build_model_and_tokenizer(config)+ accelerator2 = Accelerator(project_dir=output_dir)+ optimizer2 = torch.optim.AdamW(+ [p for p in model2.parameters() if p.requires_grad], lr=1e-4+ )+ model2, optimizer2 = accelerator2.prepare(model2, optimizer2)+ step = load_checkpoint(accelerator2, checkpoint_dir)+ assert step == 1+++def test_merge_lora_produces_deployable_model(tmp_path):+ from accelerate import Accelerator++ from omnivoice.cli.merge_lora import main as merge_main+ from omnivoice.models.omnivoice import OmniVoice+ from omnivoice.training.builder import build_model_and_tokenizer+ from omnivoice.training.checkpoint import save_checkpoint++ output_dir = str(tmp_path / "exp")+ config = _lora_config(output_dir)+ model, tokenizer = build_model_and_tokenizer(config)+ accelerator = Accelerator(project_dir=output_dir)+ os.makedirs(output_dir, exist_ok=True)+ optimizer = torch.optim.AdamW(+ [p for p in model.parameters() if p.requires_grad], lr=1e-4+ )+ model, optimizer = accelerator.prepare(model, optimizer)+ save_checkpoint(accelerator, model, tokenizer, output_dir, step=1, keep_last_n=-1)+ checkpoint_dir = os.path.join(output_dir, "checkpoint-1")++ merged_dir = str(tmp_path / "merged")+ argv = [+ "omnivoice-merge-lora",+ "--base_model",+ MODEL_PATH,+ "--lora_adapter",+ checkpoint_dir,+ "--output_dir",+ merged_dir,+ ]+ import sys++ old_argv = sys.argv+ sys.argv = argv+ try:+ merge_main()+ finally:+ sys.argv = old_argv++ merged_files = set(os.listdir(merged_dir))+ assert {"config.json", "model.safetensors", "audio_tokenizer"} <= merged_files++ # Merged model loads and runs like any plain OmniVoice checkpoint.+ reloaded = OmniVoice.from_pretrained(+ merged_dir, device_map="cpu", dtype=torch.float32+ )+ assert type(reloaded) is OmniVoice