Skip to content

SOTA Open Source TTS

Update lora document

Lengyueb5a3f899eeec
diff --git a/docs/en/finetune.md b/docs/en/finetune.md
index 7d51571..71fe057 100644
--- a/docs/en/finetune.md
+++ b/docs/en/finetune.md
@@ -166,11 +166,14 @@ huggingface-cli download fishaudio/speech-lm-v1 text2semantic-400m-v0.2-4k.pth -
Finally, you can start the fine-tuning by running the following command:
```bash
-python fish_speech/train.py --config-name text2semantic_finetune_spk
+python fish_speech/train.py --config-name text2semantic_finetune
```
+!!! info
+ If you want to use lora, please use `--config-name text2semantic_finetune_lora` to start fine-tuning.
+
!!! note
- You can modify the training parameters such as `batch_size`, `gradient_accumulation_steps`, etc. to fit your GPU memory by modifying `fish_speech/configs/text2semantic_finetune_spk.yaml`.
+ You can modify the training parameters such as `batch_size`, `gradient_accumulation_steps`, etc. to fit your GPU memory by modifying `fish_speech/configs/text2semantic_finetune.yaml`.
After training is complete, you can refer to the [inference](inference.md) section, and use `--speaker SPK1` to generate speech.
diff --git a/docs/en/index.md b/docs/en/index.md
index de9a7a6..52c0833 100644
--- a/docs/en/index.md
+++ b/docs/en/index.md
@@ -33,6 +33,7 @@ pip3 install -e .
## Changelog
+- 2023/12/28: Added `lora` fine-tuning support.
- 2023/12/27: Add `gradient checkpointing`, `causual sampling`, and `flash-attn` support.
- 2023/12/19: Updated webui and HTTP API.
- 2023/12/18: Updated fine-tuning documentation and related examples.
diff --git a/docs/en/inference.md b/docs/en/inference.md
index db10340..4cd83ea 100644
--- a/docs/en/inference.md
+++ b/docs/en/inference.md
@@ -51,6 +51,10 @@ This command will create a `codes_N` file in the working directory, where N is a
!!! info
For GPUs that do not support bf16, you may need to use the `--half` parameter.
+!!! warning
+ If you are using your own fine-tuned model, please be sure to carry the `--speaker` parameter to ensure the stability of pronunciation.
+ If you are using lora, please use `--config-name text2semantic_finetune_lora` to load the model.
+
### 3. Generate vocals from semantic tokens:
```bash
python tools/vqgan/inference.py \
diff --git a/docs/zh/finetune.md b/docs/zh/finetune.md
index f4999fe..115a978 100644
--- a/docs/zh/finetune.md
+++ b/docs/zh/finetune.md
@@ -186,11 +186,14 @@ HF_ENDPOINT=https://hf-mirror.com huggingface-cli download fishaudio/speech-lm-v
最后, 你可以运行以下命令来启动微调:
```bash
-python fish_speech/train.py --config-name text2semantic_finetune_spk
+python fish_speech/train.py --config-name text2semantic_finetune
```
!!! note
- 你可以通过修改 `fish_speech/configs/text2semantic_finetune_spk.yaml` 来修改训练参数如 `batch_size`, `gradient_accumulation_steps` 等, 来适应你的显存.
+ 如果你想使用 lora, 请使用 `--config-name text2semantic_finetune_lora` 来启动微调.
+
+!!! note
+ 你可以通过修改 `fish_speech/configs/text2semantic_finetune.yaml` 来修改训练参数如 `batch_size`, `gradient_accumulation_steps` 等, 来适应你的显存.
训练结束后, 你可以参考 [推理](inference.md) 部分, 并携带 `--speaker SPK1` 参数来测试你的模型.
diff --git a/docs/zh/index.md b/docs/zh/index.md
index 010fec0..9b43b34 100644
--- a/docs/zh/index.md
+++ b/docs/zh/index.md
@@ -33,6 +33,7 @@ pip3 install -e .
## 更新日志
+- 2023/12/28: 添加了 `lora` 微调支持.
- 2023/12/27: 添加了 `gradient checkpointing`, `causual sampling` 和 `flash-attn` 支持.
- 2023/12/19: 更新了 Webui 和 HTTP API.
- 2023/12/18: 更新了微调文档和相关例子.
diff --git a/docs/zh/inference.md b/docs/zh/inference.md
index c518ac1..80b9cc0 100644
--- a/docs/zh/inference.md
+++ b/docs/zh/inference.md
@@ -57,7 +57,8 @@ python tools/llama/generate.py \
对于不支持 bf16 的 GPU, 你可能需要使用 `--half` 参数.
!!! warning
- 如果你在使用自己微调的模型, 请务必携带 `--speaker` 参数来保证发音的稳定性.
+ 如果你在使用自己微调的模型, 请务必携带 `--speaker` 参数来保证发音的稳定性.
+ 如果你使用了 lora, 请使用 `--config-name text2semantic_finetune_lora` 来加载模型.
### 3. 从语义 token 生成人声:
```bash
diff --git a/pyproject.toml b/pyproject.toml
index eca2885..a688ae9 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -34,6 +34,7 @@ dependencies = [
"zibai-server>=0.9.0",
"loguru>=0.6.0",
"WeTextProcessing>=0.1.10",
+ "loralib>=0.1.2"
]
[project.optional-dependencies]