Skip to content

High-Quality Voice Cloning TTS for 600+ Languages

Python

feat: add optional LoRA finetuning support via PEFT (#243)

Kawshik Kumar Paul38e992bc60f8
diff --git a/docs/lora_finetuning.md b/docs/lora_finetuning.md
new file mode 100644
index 0000000..1124f3f
--- /dev/null
+++ b/docs/lora_finetuning.md
@@ -0,0 +1,194 @@
+# LoRA Finetuning
+
+In addition to full finetuning (see [training.md](training.md)), OmniVoice supports parameter-efficient finetuning via [LoRA](https://arxiv.org/abs/2106.09685) (Low-Rank Adaptation), using [PEFT](https://github.com/huggingface/peft). LoRA freezes the pretrained LLM backbone and trains small low-rank adapter matrices instead — typically **3-5% of parameters trainable** — which means:
+
+- Much lower optimizer memory (Adam keeps 2 extra tensors per trainable parameter).
+- Much smaller adapter checkpoints to ship (tens of MB vs. the full model size).
+- The original pretrained weights stay untouched on disk; the adapter can be dropped to recover the original model exactly.
+
+LoRA is **opt-in** via config — with `use_lora: false` (the default), training behaves exactly as documented in [training.md](training.md).
+
+## Installation
+
+```bash
+pip install "omnivoice[lora]"
+# or, from a source checkout:
+pip install peft
+```
+
+## What Gets Adapted
+
+| Config field | Applies to | Why |
+|---|---|---|
+| `lora_target_modules` | LLM attention/MLP projections (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` by default) | This is where the pretrained language/text-conditioning knowledge lives — the standard set of layers LoRA adapts for transformer LLMs. |
+| `lora_modules_to_save` | `audio_embeddings`, `audio_heads` by default | These are OmniVoice-specific audio token I/O layers, not part of the pretrained LLM. There's no pretrained knowledge in them to preserve via low-rank adapters, so they're kept **fully trainable** instead and saved alongside the adapter. |
+
+Everything else (the LLM's other weights, layernorms, etc.) stays frozen, exactly as loaded from the base checkpoint.
+
+## Configuration
+
+Start from [examples/config/train_config_finetune_lora.json](../examples/config/train_config_finetune_lora.json) — identical to [train_config_finetune_sdpa.json](../examples/config/train_config_finetune_sdpa.json) plus the LoRA fields:
+
+```json
+{
+ "init_from_checkpoint": "k2-fsa/OmniVoice",
+
+ "use_lora": true,
+ "lora_r": 16,
+ "lora_alpha": 32,
+ "lora_dropout": 0.05,
+ "lora_bias": "none",
+ "lora_target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
+ "lora_modules_to_save": ["audio_embeddings", "audio_heads"],
+
+ "learning_rate": 1e-4,
+ "attn_implementation": "sdpa"
+}
+```
+
+| Field | Description | Default |
+|---|---|---|
+| `use_lora` | Enable LoRA finetuning instead of full finetuning | `false` |
+| `lora_r` | LoRA rank — higher = more capacity, more trainable params | `16` |
+| `lora_alpha` | LoRA scaling factor (effective scale is `lora_alpha / lora_r`) | `32` |
+| `lora_dropout` | Dropout applied inside the LoRA adapter path | `0.05` |
+| `lora_bias` | Which biases to train: `"none"`, `"all"`, or `"lora_only"` | `"none"` |
+| `lora_target_modules` | LLM submodule names to attach LoRA adapters to | attention + MLP projections |
+| `lora_modules_to_save` | Non-LLM modules trained in full instead of via LoRA | `audio_embeddings`, `audio_heads` |
+
+A higher `learning_rate` than full finetuning (`1e-4` vs. `5e-5`) is typically used since the adapter matrices are randomly initialized and trained from scratch, rather than finetuning already-converged pretrained weights — a higher LR converges faster without the instability risk full finetuning has at that LR.
+
+## Launching Training
+
+```bash
+accelerate launch \
+ --gpu_ids "0,1" \
+ --num_processes 2 \
+ -m omnivoice.cli.train \
+ --train_config examples/config/train_config_finetune_lora.json \
+ --data_config examples/config/data_config_finetune.json \
+ --output_dir exp/omnivoice_finetune_lora
+```
+
+Or run the full example pipeline (tokenization + training):
+
+```bash
+bash examples/run_finetune_lora.sh
+```
+
+On startup you'll see a line like:
+
+```text
+trainable params: 19,316,736 || all params: 631,894,016 || trainable%: 3.0570
+```
+
+confirming only the adapters plus `audio_embeddings`/`audio_heads` are being trained.
+
+## How Many Checkpoints Get Saved
+
+Same three fields as full finetuning govern this:
+
+| Field | Effect |
+|---|---|
+| `steps` | Total training steps |
+| `save_steps` | A checkpoint is saved every `save_steps` steps |
+| `keep_last_n_checkpoints` | `-1` keeps every checkpoint; a positive `N` rotates old ones out, keeping only the most recent `N` |
+
+Number of checkpoints saved = `steps / save_steps` (when `steps` is a multiple of `save_steps`), capped at `keep_last_n_checkpoints` if positive.
+
+## Resuming Training
+
+Identical mechanism to full finetuning — set `resume_from_checkpoint`:
+
+```json
+{
+ "resume_from_checkpoint": "exp/omnivoice_finetune_lora/checkpoint-500"
+}
+```
+
+Each run rebuilds the same LoRA-wrapped model architecture from `init_from_checkpoint` plus the `lora_*` config fields, then restores optimizer/scheduler/RNG state and adapter weights from the checkpoint on top of it. Keep `init_from_checkpoint` and the `lora_*` fields unchanged across a resume — a mismatch will surface as a shape/key error when `accelerate` restores state.
+
+## Adapter Inference
+
+Apply a trained adapter on top of the base model at inference time — merged into the model in-memory (not written to disk), so generation uses the regular `OmniVoice.generate()` path unmodified:
+
+```bash
+omnivoice-infer \
+ --model k2-fsa/OmniVoice \
+ --lora_adapter exp/omnivoice_finetune_lora/checkpoint-500 \
+ --text "Hello, this is a text for text-to-speech." \
+ --ref_audio ref.wav --ref_text "Reference transcript." \
+ --output out.wav
+```
+
+`--lora_adapter` accepts any directory containing `adapter_config.json` + `adapter_model.safetensors` — i.e. any LoRA training checkpoint directory.
+
+## Adapter Merging
+
+To deploy with **no PEFT dependency at inference time**, merge the adapter into the base model once and save the result as a normal, standalone OmniVoice checkpoint:
+
+```bash
+omnivoice-merge-lora \
+ --base_model k2-fsa/OmniVoice \
+ --lora_adapter exp/omnivoice_finetune_lora/checkpoint-500 \
+ --output_dir exp/omnivoice_finetune_lora/merged
+```
+
+This loads the base model, merges the LoRA deltas into its weights, and writes a complete, self-contained OmniVoice directory (model weights, tokenizer, and audio tokenizer).
+
+## Using the Merged Model
+
+The merged directory is a normal OmniVoice checkpoint — use it exactly like any other:
+
+```bash
+omnivoice-infer \
+ --model exp/omnivoice_finetune_lora/merged \
+ --text "Hello, this is a text for text-to-speech." \
+ --instruct "male, British accent" \
+ --output out.wav
+```
+
+```python
+from omnivoice.models.omnivoice import OmniVoice
+
+model = OmniVoice.from_pretrained("exp/omnivoice_finetune_lora/merged", device_map="cuda:0")
+audios = model.generate(text="Hello world", language="en")
+```
+
+The merged model is saved in whatever dtype the base model was loaded in when merging (`float32` by default, since `omnivoice-merge-lora` loads the base model with `dtype=torch.float32`). Pass `--dtype fp32` / `dtype=torch.float32` at inference to match exactly, or `fp16`/`bf16` for less VRAM and faster inference — `from_pretrained` casts down automatically, and quality differences are usually negligible.
+
+## Checkpoint Structure
+
+A LoRA training checkpoint (`exp/.../checkpoint-<step>/`) contains two independent sets of files:
+
+```text
+checkpoint-500/
+├── adapter_config.json # PEFT LoRA config (small)
+├── adapter_model.safetensors # LoRA weights + audio_embeddings/audio_heads (small)
+├── tokenizer.json # text tokenizer, saved for adapter/merge use
+├── tokenizer_config.json
+├── model.safetensors # accelerate's full training state (base + adapter weights)
+├── optimizer.bin # optimizer state, for resuming
+├── scheduler.bin # LR scheduler state, for resuming
+└── random_states_0.pkl # RNG state, for resuming
+```
+
+- **For adapter inference / merging**, only `adapter_config.json` + `adapter_model.safetensors` (+ tokenizer) are used.
+- **For resuming training**, `accelerate`'s own state files (`model.safetensors`, `optimizer.bin`, `scheduler.bin`, `random_states_*.pkl`) are used — these include the full (frozen + trainable) model state, since `accelerate`'s checkpointing does not distinguish frozen from trainable parameters. This makes `checkpoint-*/` directories larger than the adapter alone, but keeps resuming exactly as reliable as full finetuning's. If you only need the adapter for deployment, you can delete everything except `adapter_config.json`/`adapter_model.safetensors`/the tokenizer files once training is done.
+
+## Troubleshooting
+
+**`ModuleNotFoundError: No module named 'peft'`**
+Run `pip install peft` (or `pip install "omnivoice[lora]"`).
+
+**Resume fails with a shape/key mismatch error**
+`resume_from_checkpoint` must be resumed with the *same* `init_from_checkpoint`, `lora_r`, `lora_alpha`, `lora_target_modules`, and `lora_modules_to_save` as the run that produced the checkpoint — these determine the model architecture `accelerate`'s state restore expects to find.
+
+**`omnivoice-merge-lora` / `--lora_adapter` can't find the adapter**
+Point it at a checkpoint *directory*, not a specific file — it must contain `adapter_config.json`.
+
+**Training loss looks the same as a frozen model / doesn't move**
+Check the `trainable params: ...` line printed at startup. If it's `0` or doesn't include LoRA layers, verify `use_lora: true` is set in the config passed to `--train_config`, and that `lora_target_modules` names match submodules in your `llm_name_or_path`'s architecture.
+
+**Checkpoint saves or merging are unexpectedly slow**
+Each checkpoint writes a full base-model-sized file (accelerate's training state — see [Checkpoint Structure](#checkpoint-structure)), and merging both reads and writes one. If disk space is tight or I/O is contended, these can take much longer than expected; check available disk space first.
diff --git a/docs/training.md b/docs/training.md
index 9339e51..d89dfc9 100644
--- a/docs/training.md
+++ b/docs/training.md
@@ -94,6 +94,10 @@ To start training from a pretrained OmniVoice checkpoint (for fine-tuning):
}
```
+## LoRA Finetuning
+
+For parameter-efficient finetuning, set `"use_lora": true` in your training config to train low-rank adapters instead of the full model. See [lora_finetuning.md](lora_finetuning.md) for configuration, adapter inference, and merging back into a standalone checkpoint.
+
## Monitoring
Training logs to TensorBoard:
diff --git a/examples/README.md b/examples/README.md
index 00f6485..566a5ec 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -6,6 +6,7 @@ This directory contains scripts and configs for training, fine-tuning, and evalu
|---|---|---|
| Training from scratch | [run_emilia.sh](run_emilia.sh) | Full pipeline on the Emilia dataset (data check, tokenization, training) |
| Fine-tuning | [run_finetune.sh](run_finetune.sh) | Fine-tune from a pretrained checkpoint using your own JSONL data |
+| LoRA fine-tuning | [run_finetune_lora.sh](run_finetune_lora.sh) | Parameter-efficient finetuning via PEFT LoRA adapters — see [../docs/lora_finetuning.md](../docs/lora_finetuning.md) |
| Evaluation | [run_eval.sh](run_eval.sh) | Evaluate WER, speaker similarity, and UTMOS on standard test sets |
---
diff --git a/examples/config/train_config_finetune_lora.json b/examples/config/train_config_finetune_lora.json
new file mode 100644
index 0000000..3c02600
--- /dev/null
+++ b/examples/config/train_config_finetune_lora.json
@@ -0,0 +1,52 @@
+{
+ "llm_name_or_path": "Qwen/Qwen3-0.6B",
+ "audio_vocab_size": 1025,
+ "audio_mask_id": 1024,
+ "num_audio_codebook": 8,
+
+ "audio_codebook_weights": [8, 8, 6, 6, 4, 4, 2, 2],
+ "drop_cond_ratio": 0.1,
+ "prompt_ratio_range": [0.0, 0.3],
+ "mask_ratio_range": [0.0, 1.0],
+ "language_ratio": 0.8,
+ "use_pinyin_ratio": 0.0,
+ "instruct_ratio": 0.0,
+ "only_instruct_ratio": 0.0,
+
+ "resume_from_checkpoint": null,
+ "init_from_checkpoint": "k2-fsa/OmniVoice",
+
+ "use_lora": true,
+ "lora_r": 16,
+ "lora_alpha": 32,
+ "lora_dropout": 0.05,
+ "lora_bias": "none",
+ "lora_target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
+ "lora_modules_to_save": ["audio_embeddings", "audio_heads"],
+
+ "learning_rate": 1e-4,
+ "weight_decay": 0.01,
+ "max_grad_norm": 1.0,
+ "steps": 5000,
+ "seed": 42,
+ "warmup_type": "ratio",
+ "warmup_ratio": 0.01,
+ "warmup_steps": 0,
+
+ "attn_implementation": "sdpa",
+ "max_sample_tokens": 2000,
+ "min_sample_tokens": 50,
+ "max_batch_size": 64,
+
+ "batch_tokens": 8192,
+ "gradient_accumulation_steps": 1,
+ "num_workers": 2,
+
+ "mixed_precision": "bf16",
+ "allow_tf32": true,
+
+ "logging_steps": 50,
+ "eval_steps": 500,
+ "save_steps": 500,
+ "keep_last_n_checkpoints": -1
+}
diff --git a/examples/run_finetune_lora.sh b/examples/run_finetune_lora.sh
new file mode 100755
index 0000000..5bf200a
--- /dev/null
+++ b/examples/run_finetune_lora.sh
@@ -0,0 +1,85 @@
+#!/bin/bash
+
+# This script demonstrates how to LoRA fine-tune OmniVoice from a JSONL manifest.
+# See ../docs/lora_finetuning.md for the full guide (why each step is required,
+# adapter inference, and adapter merging for deployment).
+
+set -euo pipefail
+
+stage=0
+stop_stage=1
+
+# ====== Modify as needed ======
+# GPUs to use
+GPU_IDS="0,1"
+NUM_GPUS=2
+
+# Path to your input JSONL file
+# (each line: {"id": ..., "audio_path": ..., "text": ..., "language_id": ...})
+TRAIN_JSONL="data/my_data_train.jsonl"
+
+# Path to your dev JSONL file. Set to empty string to skip dev set.
+DEV_JSONL="data/my_data_dev.jsonl"
+
+# Directory to write tokenized WebDataset shards
+TOKEN_DIR="data/finetune/tokens"
+
+# Audio tokenizer model (HuggingFace repo or local path)
+TOKENIZER_PATH="eustlb/higgs-audio-v2-tokenizer"
+
+# LoRA training config file
+TRAIN_CONFIG="config/train_config_finetune_lora.json"
+
+# Data config file
+data_config="config/data_config_finetune.json"
+
+# Output directory for the LoRA adapter checkpoints
+OUTPUT_DIR="exp/omnivoice_finetune_lora"
+# =================================
+
+export PYTHONPATH="$(cd "$(dirname "$0")/.." && pwd):${PYTHONPATH:-}"
+
+
+# Stage 0: Tokenize audio into WebDataset shards
+if [ $stage -le 0 ] && [ $stop_stage -ge 0 ]; then
+ echo "Stage 0: Tokenizing audio"
+
+ for split_jsonl_path in ${TRAIN_JSONL} ${DEV_JSONL}; do
+ if [ -z "${split_jsonl_path}" ]; then
+ continue
+ fi
+
+ if [ "${split_jsonl_path}" = "${TRAIN_JSONL}" ]; then
+ split="train"
+ else
+ split="dev"
+ fi
+
+ echo " Tokenizing ${split} from ${split_jsonl_path}"
+
+ CUDA_VISIBLE_DEVICES=${GPU_IDS} \
+ python -m omnivoice.scripts.extract_audio_tokens \
+ --input_jsonl "${split_jsonl_path}" \
+ --tar_output_pattern "${TOKEN_DIR}/${split}/audios/shard-%06d.tar" \
+ --jsonl_output_pattern "${TOKEN_DIR}/${split}/txts/shard-%06d.jsonl" \
+ --tokenizer_path "${TOKENIZER_PATH}" \
+ --nj_per_gpu 3 \
+ --shuffle True
+
+ echo " Done. Manifest written to ${TOKEN_DIR}/${split}/data.lst"
+ done
+fi
+
+
+# Stage 1: LoRA fine-tune
+if [ $stage -le 1 ] && [ $stop_stage -ge 1 ]; then
+ echo "Stage 1: LoRA fine-tuning"
+
+ accelerate launch \
+ --gpu_ids "${GPU_IDS}" \
+ --num_processes ${NUM_GPUS} \
+ -m omnivoice.cli.train \
+ --train_config ${TRAIN_CONFIG} \
+ --data_config ${data_config} \
+ --output_dir ${OUTPUT_DIR}
+fi
diff --git a/omnivoice/cli/infer.py b/omnivoice/cli/infer.py
index ce4657d..2d903cc 100644
--- a/omnivoice/cli/infer.py
+++ b/omnivoice/cli/infer.py
@@ -53,6 +53,14 @@ def get_parser() -> argparse.ArgumentParser:
required=True,
help="Output WAV file path.",
)
+ parser.add_argument(
+ "--lora_adapter",
+ type=str,
+ default=None,
+ help="Path to a LoRA adapter directory (e.g. a LoRA training "
+ "checkpoint) to apply on top of --model. Merged in-memory before "
+ "generation.",
+ )
# Voice cloning
parser.add_argument(
"--ref_audio",
@@ -122,6 +130,12 @@ def main():
args.model, device_map=device, dtype=torch.float16
)
+ if args.lora_adapter:
+ from omnivoice.utils.lora import load_lora_adapter
+
+ logging.info(f"Applying LoRA adapter from {args.lora_adapter} ...")
+ model = load_lora_adapter(model, args.lora_adapter)
+
logging.info(f"Generating audio for: {args.text[:80]}...")
audios = model.generate(
text=args.text,
diff --git a/omnivoice/cli/merge_lora.py b/omnivoice/cli/merge_lora.py
new file mode 100644
index 0000000..6e5d86f
--- /dev/null
+++ b/omnivoice/cli/merge_lora.py
@@ -0,0 +1,123 @@
+#!/usr/bin/env python3
+# Copyright 2026 Xiaomi Corp. (authors: Han Zhu)
+#
+# See ../../LICENSE for clarification regarding multiple authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Merge a LoRA adapter into its base model for normal OmniVoice deployment.
+
+Loads the base model, merges the LoRA adapter weights into it, and saves the
+result as a plain HuggingFace-format OmniVoice checkpoint (config.json +
+model.safetensors + tokenizer + audio_tokenizer) that can be loaded directly
+with ``OmniVoice.from_pretrained()`` — no PEFT dependency needed at inference
+time.
+
+Usage:
+ omnivoice-merge-lora \\
+ --base_model /path/to/OmniVoice \\
+ --lora_adapter exp/omnivoice_lora/checkpoint-5000 \\
+ --output_dir exp/omnivoice_lora/merged
+"""
+
+import argparse
+import logging
+import os
+import shutil
+
+import torch
+from transformers import AutoTokenizer
+
+from omnivoice.models.omnivoice import OmniVoice, _resolve_model_path
+from omnivoice.utils.lora import load_lora_adapter
+
+logger = logging.getLogger(__name__)
+
+# Files that are part of a deployable OmniVoice directory but are not part of
+# the model's state dict, so save_pretrained() does not (re)write them.
+_EXTRA_DEPLOY_PATHS = ["audio_tokenizer", "chat_template.jinja"]
+
+
+def get_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(
+ description="Merge a LoRA adapter into its base OmniVoice model",
+ formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+ )
+ parser.add_argument(
+ "--base_model",
+ type=str,
+ required=True,
+ help="Base model checkpoint path (local dir or HuggingFace repo id) "
+ "the LoRA adapter was trained from (i.e. its init_from_checkpoint).",
+ )
+ parser.add_argument(
+ "--lora_adapter",
+ type=str,
+ required=True,
+ help="Path to the LoRA adapter directory (e.g. a LoRA training "
+ "checkpoint) containing adapter_config.json.",
+ )
+ parser.add_argument(
+ "--output_dir",
+ type=str,
+ required=True,
+ help="Directory to write the merged, deployable OmniVoice model to.",
+ )
+ return parser
+
+
+def main():
+ logging.basicConfig(
+ format="%(asctime)s %(levelname)s [%(filename)s:%(lineno)d] %(message)s",
+ level=logging.INFO,
+ )
+ args = get_parser().parse_args()
+
+ logger.info(f"Loading base model from {args.base_model} ...")
+ model = OmniVoice.from_pretrained(args.base_model, train=True, dtype=torch.float32)
+
+ merged = load_lora_adapter(model, args.lora_adapter, merge=True)
+
+ os.makedirs(args.output_dir, exist_ok=True)
+ logger.info(f"Saving merged model to {args.output_dir} ...")
+ merged.save_pretrained(args.output_dir)
+
+ # Prefer the tokenizer saved alongside the adapter (training checkpoints
+ # save it there); it is identical to the base model's unless new special
+ # tokens were added, in which case it is the correct one to keep.
+ tokenizer_source = (
+ args.lora_adapter
+ if os.path.exists(os.path.join(args.lora_adapter, "tokenizer_config.json"))
+ else args.base_model
+ )
+ tokenizer = AutoTokenizer.from_pretrained(tokenizer_source)
+ tokenizer.save_pretrained(args.output_dir)
+
+ # Copy over deployment files that aren't part of the model state dict
+ # (audio tokenizer, chat template) so the output directory is a
+ # self-contained, directly loadable OmniVoice checkpoint.
+ resolved_base = _resolve_model_path(args.base_model)
+ for name in _EXTRA_DEPLOY_PATHS:
+ src = os.path.join(resolved_base, name)
+ dst = os.path.join(args.output_dir, name)
+ if os.path.exists(src) and not os.path.exists(dst):
+ if os.path.isdir(src):
+ shutil.copytree(src, dst)
+ else:
+ shutil.copy2(src, dst)
+
+ logger.info(f"Merged model saved to {args.output_dir}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/omnivoice/training/builder.py b/omnivoice/training/builder.py
index 6a33431..72c50ff 100644
--- a/omnivoice/training/builder.py
+++ b/omnivoice/training/builder.py
@@ -33,6 +33,7 @@ Key functions:
"""
import logging
+import os
from functools import partial
from typing import Tuple
@@ -125,9 +126,37 @@ def build_model_and_tokenizer(
model.config.bos_token_id = tokenizer.bos_token_id
model.config.eos_token_id = tokenizer.eos_token_id
+ # 5. Wrap with LoRA adapters (optional)
+ if config.use_lora:
+ model = _apply_lora(model, config)
+
return model, tokenizer
+def _apply_lora(model: OmniVoice, config: TrainingConfig) -> OmniVoice:
+ """Wrap the model with PEFT LoRA adapters.
+
+ ``lora_target_modules`` (LLM attention/MLP projections) are trained as
+ low-rank adapters; ``lora_modules_to_save`` (the audio embedding and
+ output head) are OmniVoice-specific I/O layers with no LLM-pretrained
+ knowledge to preserve, so they are kept fully trainable instead.
+ """
+ from peft import LoraConfig, get_peft_model
+
+ lora_config = LoraConfig(
+ r=config.lora_r,
+ lora_alpha=config.lora_alpha,
+ lora_dropout=config.lora_dropout,
+ bias=config.lora_bias,
+ target_modules=config.lora_target_modules,
+ modules_to_save=config.lora_modules_to_save,
+ )
+ model = get_peft_model(model, lora_config)
+ if int(os.environ.get("LOCAL_RANK", "0")) == 0:
+ model.print_trainable_parameters()
+ return model
+
+
def build_dataloaders(
config: TrainingConfig, tokenizer: AutoTokenizer
) -> Tuple[DataLoader, DataLoader]:
diff --git a/omnivoice/training/config.py b/omnivoice/training/config.py
index 49cd880..9756451 100644
--- a/omnivoice/training/config.py
+++ b/omnivoice/training/config.py
@@ -55,6 +55,30 @@ class TrainingConfig:
resume_from_checkpoint: Optional[str] = None
init_from_checkpoint: Optional[str] = None
+ # LoRA (PEFT) Settings
+ use_lora: bool = False
+ lora_r: int = 16
+ lora_alpha: int = 32
+ lora_dropout: float = 0.05
+ lora_bias: str = "none"
+ lora_target_modules: List[str] = field(
+ default_factory=lambda: [
+ "q_proj",
+ "k_proj",
+ "v_proj",
+ "o_proj",
+ "gate_proj",
+ "up_proj",
+ "down_proj",
+ ]
+ )
+ # Non-LLM modules trained in full rather than via low-rank adapters,
+ # since they are OmniVoice-specific audio I/O layers, not general LLM
+ # knowledge, and are small relative to the LLM backbone.
+ lora_modules_to_save: List[str] = field(
+ default_factory=lambda: ["audio_embeddings", "audio_heads"]
+ )
+
# Training Hyperparams
learning_rate: float = 1e-4
weight_decay: float = 0.01
diff --git a/omnivoice/training/trainer.py b/omnivoice/training/trainer.py
index ec015be..9ea53e6 100644
--- a/omnivoice/training/trainer.py
+++ b/omnivoice/training/trainer.py
@@ -164,8 +164,11 @@ class OmniTrainer:
def create_optimizer_and_scheduler(self):
"""Default AdamW + configurable LR Scheduler."""
+ # Only trainable parameters (all of them for full fine-tuning; LoRA
+ # adapters + modules_to_save when config.use_lora froze the rest).
+ trainable_params = [p for p in self.model.parameters() if p.requires_grad]
optimizer = torch.optim.AdamW(
- self.model.parameters(),
+ trainable_params,
lr=self.config.learning_rate,
weight_decay=self.config.weight_decay,
)
diff --git a/omnivoice/utils/lora.py b/omnivoice/utils/lora.py
new file mode 100644
index 0000000..905128e
--- /dev/null
+++ b/omnivoice/utils/lora.py
@@ -0,0 +1,59 @@
+#!/usr/bin/env python3
+# Copyright 2026 Xiaomi Corp. (authors: Han Zhu)
+#
+# See ../../LICENSE for clarification regarding multiple authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""LoRA adapter loading shared by inference and merge CLIs.
+
+Used by ``omnivoice.cli.infer`` (``--lora_adapter``) and
+``omnivoice.cli.merge_lora``.
+"""
+
+import logging
+from typing import Union
+
+from omnivoice.models.omnivoice import OmniVoice
+
+logger = logging.getLogger(__name__)
+
+
+def load_lora_adapter(
+ model: OmniVoice, adapter_path: str, merge: bool = True
+) -> Union[OmniVoice, "peft.PeftModel"]: # noqa: F821
+ """Attach a trained LoRA adapter to a loaded ``OmniVoice`` model.
+
+ Args:
+ model: An ``OmniVoice`` model, as returned by
+ :meth:`OmniVoice.from_pretrained`.
+ adapter_path: Path to a LoRA adapter directory containing
+ ``adapter_config.json`` and ``adapter_model.safetensors``, e.g.
+ a LoRA training checkpoint directory.
+ merge: If ``True`` (default), merge the adapter weights into
+ ``model`` in-memory and return the plain ``OmniVoice`` model, so
+ its custom methods (``generate``, ``create_voice_clone_prompt``,
+ etc.) keep working unmodified. If ``False``, return the
+ ``peft.PeftModel`` wrapper with the adapter kept separate.
+
+ Returns:
+ The merged ``OmniVoice`` model (``merge=True``) or the wrapping
+ ``PeftModel`` (``merge=False``).
+ """
+ from peft import PeftModel
+
+ logger.info("Loading LoRA adapter from %s", adapter_path)
+ peft_model = PeftModel.from_pretrained(model, adapter_path)
+ if merge:
+ return peft_model.merge_and_unload()
+ return peft_model
diff --git a/pyproject.toml b/pyproject.toml
index bd7f04f..b93124b 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -59,10 +59,15 @@ eval = [
"unidecode", # Unicode normalization
]
+lora = [
+ "peft>=0.20.0", # LoRA fine-tuning support
+]
+
[project.scripts]
omnivoice-infer = "omnivoice.cli.infer:main"
omnivoice-infer-batch = "omnivoice.cli.infer_batch:main"
omnivoice-demo = "omnivoice.cli.demo:main"
+omnivoice-merge-lora = "omnivoice.cli.merge_lora:main"
[project.urls]
Homepage = "https://github.com/k2-fsa/OmniVoice"
diff --git a/tests/test_lora.py b/tests/test_lora.py
new file mode 100644
index 0000000..acb6762
--- /dev/null
+++ b/tests/test_lora.py
@@ -0,0 +1,225 @@
+#!/usr/bin/env python3
+# Copyright 2026 Xiaomi Corp. (authors: Han Zhu)
+#
+# See ../LICENSE for clarification regarding multiple authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for LoRA fine-tuning support (config, model wrapping, checkpoint
+save/resume, and adapter merging).
+
+Requires a local OmniVoice model directory, set via the
+``OMNIVOICE_TEST_MODEL_PATH`` environment variable. Tests are skipped if it
+is not set or does not exist, since they load the actual pretrained model.
+"""
+
+import json
+import os
+
+import pytest
+import torch
+
+MODEL_PATH = os.environ.get("OMNIVOICE_TEST_MODEL_PATH")
+
+pytestmark = pytest.mark.skipif(
+ not MODEL_PATH or not os.path.isdir(MODEL_PATH),
+ reason="Set OMNIVOICE_TEST_MODEL_PATH to a local OmniVoice model directory to run these tests.",
+)
+
+
+def _lora_config(output_dir, **overrides):
+ from omnivoice.training.config import TrainingConfig
+
+ kwargs = {
+ "llm_name_or_path": MODEL_PATH,
+ "init_from_checkpoint": MODEL_PATH,
+ "attn_implementation": "sdpa",
+ "use_lora": True,
+ "lora_r": 4,
+ "lora_alpha": 8,
+ "output_dir": output_dir,
+ "steps": 2,
+ }
+ kwargs.update(overrides)
+ return TrainingConfig(**kwargs)
+
+
+def test_training_config_lora_defaults():
+ from omnivoice.training.config import TrainingConfig
+
+ config = TrainingConfig()
+ assert config.use_lora is False
+ assert config.lora_r == 16
+ assert "q_proj" in config.lora_target_modules
+ assert "audio_embeddings" in config.lora_modules_to_save
+ assert "audio_heads" in config.lora_modules_to_save
+
+
+def test_training_config_lora_json_roundtrip(tmp_path):
+ from omnivoice.training.config import TrainingConfig
+
+ config = _lora_config(str(tmp_path))
+ json_path = tmp_path / "train_config.json"
+ config.save_to_json(str(json_path))
+
+ with open(json_path) as f:
+ data = json.load(f)
+ assert data["use_lora"] is True
+ assert data["lora_r"] == 4
+
+ reloaded = TrainingConfig.from_json(str(json_path))
+ assert reloaded.use_lora is True
+ assert reloaded.lora_r == 4
+
+
+def test_build_model_with_lora_freezes_base_and_keeps_io_heads_trainable(tmp_path):
+ from omnivoice.training.builder import build_model_and_tokenizer
+
+ config = _lora_config(str(tmp_path))
+ model, _tokenizer = build_model_and_tokenizer(config)
+
+ from peft import PeftModel
+
+ assert isinstance(model, PeftModel)
+
+ trainable = {n for n, p in model.named_parameters() if p.requires_grad}
+ assert any("lora_A" in n for n in trainable)
+ assert any("lora_B" in n for n in trainable)
+ # audio_embeddings/audio_heads are modules_to_save -> fully trainable
+ assert any("audio_embeddings" in n for n in trainable)
+ assert any("audio_heads" in n for n in trainable)
+ # base LLM linear weights themselves (not the lora_A/B deltas) stay frozen
+ frozen = {n for n, p in model.named_parameters() if not p.requires_grad}
+ assert any(n.endswith("self_attn.q_proj.base_layer.weight") for n in frozen)
+
+
+def test_build_model_without_lora_is_plain_omnivoice(tmp_path):
+ from omnivoice.models.omnivoice import OmniVoice
+ from omnivoice.training.builder import build_model_and_tokenizer
+
+ config = _lora_config(str(tmp_path), use_lora=False)
+ model, _tokenizer = build_model_and_tokenizer(config)
+
+ assert type(model) is OmniVoice
+ assert all(p.requires_grad for p in model.parameters())
+
+
+def test_lora_forward_and_backward(tmp_path):
+ from omnivoice.training.builder import build_model_and_tokenizer
+
+ config = _lora_config(str(tmp_path))
+ model, _tokenizer = build_model_and_tokenizer(config)
+
+ batch, num_codebooks, seq_len, cond_len = 1, 8, 20, 10
+ input_ids = torch.randint(0, 100, (batch, num_codebooks, seq_len))
+ audio_mask = torch.zeros(batch, seq_len, dtype=torch.bool)
+ audio_mask[:, cond_len:] = True
+ labels = torch.full((batch, num_codebooks, seq_len), -100, dtype=torch.long)
+ labels[:, :, cond_len:] = torch.randint(
+ 0, 100, (batch, num_codebooks, seq_len - cond_len)
+ )
+
+ out = model(input_ids=input_ids, audio_mask=audio_mask, labels=labels)
+ assert out.loss is not None and torch.isfinite(out.loss)
+ out.loss.backward()
+
+ grad_params = [p for p in model.parameters() if p.grad is not None]
+ assert len(grad_params) > 0
+
+
+def test_lora_checkpoint_save_and_resume(tmp_path):
+ from accelerate import Accelerator
+
+ from omnivoice.training.builder import build_model_and_tokenizer
+ from omnivoice.training.checkpoint import load_checkpoint, save_checkpoint
+
+ output_dir = str(tmp_path / "exp")
+ config = _lora_config(output_dir)
+
+ model, tokenizer = build_model_and_tokenizer(config)
+ accelerator = Accelerator(project_dir=output_dir)
+ os.makedirs(output_dir, exist_ok=True)
+
+ optimizer = torch.optim.AdamW(
+ [p for p in model.parameters() if p.requires_grad], lr=1e-4
+ )
+ model, optimizer = accelerator.prepare(model, optimizer)
+
+ save_checkpoint(accelerator, model, tokenizer, output_dir, step=1, keep_last_n=-1)
+
+ checkpoint_dir = os.path.join(output_dir, "checkpoint-1")
+ saved_files = set(os.listdir(checkpoint_dir))
+ # Small PEFT adapter files, used for inference/merging.
+ assert {"adapter_config.json", "adapter_model.safetensors"} <= saved_files
+ # Full accelerator training state, used for resuming.
+ assert {"optimizer.bin"} <= saved_files
+
+ # Resuming requires rebuilding the same LoRA architecture first, then
+ # restoring accelerator state on top of it (mirrors full fine-tuning).
+ model2, _tokenizer2 = build_model_and_tokenizer(config)
+ accelerator2 = Accelerator(project_dir=output_dir)
+ optimizer2 = torch.optim.AdamW(
+ [p for p in model2.parameters() if p.requires_grad], lr=1e-4
+ )
+ model2, optimizer2 = accelerator2.prepare(model2, optimizer2)
+ step = load_checkpoint(accelerator2, checkpoint_dir)
+ assert step == 1
+
+
+def test_merge_lora_produces_deployable_model(tmp_path):
+ from accelerate import Accelerator
+
+ from omnivoice.cli.merge_lora import main as merge_main
+ from omnivoice.models.omnivoice import OmniVoice
+ from omnivoice.training.builder import build_model_and_tokenizer
+ from omnivoice.training.checkpoint import save_checkpoint
+
+ output_dir = str(tmp_path / "exp")
+ config = _lora_config(output_dir)
+ model, tokenizer = build_model_and_tokenizer(config)
+ accelerator = Accelerator(project_dir=output_dir)
+ os.makedirs(output_dir, exist_ok=True)
+ optimizer = torch.optim.AdamW(
+ [p for p in model.parameters() if p.requires_grad], lr=1e-4
+ )
+ model, optimizer = accelerator.prepare(model, optimizer)
+ save_checkpoint(accelerator, model, tokenizer, output_dir, step=1, keep_last_n=-1)
+ checkpoint_dir = os.path.join(output_dir, "checkpoint-1")
+
+ merged_dir = str(tmp_path / "merged")
+ argv = [
+ "omnivoice-merge-lora",
+ "--base_model",
+ MODEL_PATH,
+ "--lora_adapter",
+ checkpoint_dir,
+ "--output_dir",
+ merged_dir,
+ ]
+ import sys
+
+ old_argv = sys.argv
+ sys.argv = argv
+ try:
+ merge_main()
+ finally:
+ sys.argv = old_argv
+
+ merged_files = set(os.listdir(merged_dir))
+ assert {"config.json", "model.safetensors", "audio_tokenizer"} <= merged_files
+
+ # Merged model loads and runs like any plain OmniVoice checkpoint.
+ reloaded = OmniVoice.from_pretrained(
+ merged_dir, device_map="cpu", dtype=torch.float32
+ )
+ assert type(reloaded) is OmniVoice