Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
Large-Scale Data Training
游雁82e5ca37a8bd
diff --git a/docs/tutorial/README.md b/docs/tutorial/README.mdindex 74febcddf..933b611a5 100644--- a/docs/tutorial/README.md+++ b/docs/tutorial/README.md@@ -211,7 +211,7 @@ Full code ref to [finetune.sh](https://github.com/alibaba-damo-academy/FunASR/bl### Detailed Parameter Description:```shell-funasr/bin/train.py \+funasr/bin/train_ds.py \++model="${model_name_or_model_dir}" \++train_data_set_list="${train_data}" \++valid_data_set_list="${val_data}" \@@ -252,7 +252,7 @@ export CUDA_VISIBLE_DEVICES="0,1"gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')torchrun --nnodes 1 --nproc_per_node ${gpu_num} \-../../../funasr/bin/train.py ${train_args}+../../../funasr/bin/train_ds.py ${train_args}```--nnodes represents the total number of participating nodes, while --nproc_per_node indicates the number of processes running on each node.@@ -264,7 +264,7 @@ export CUDA_VISIBLE_DEVICES="0,1"gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')torchrun --nnodes 2 --node_rank 0 --nproc_per_node ${gpu_num} --master_addr=192.168.1.1 --master_port=12345 \-../../../funasr/bin/train.py ${train_args}+../../../funasr/bin/train_ds.py ${train_args}```On the worker node (assuming the IP is 192.168.1.2), you need to ensure that the MASTER_ADDR and MASTER_PORT environment variables are set to match those of the master node, and then run the same command:@@ -273,7 +273,7 @@ export CUDA_VISIBLE_DEVICES="0,1"gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')torchrun --nnodes 2 --node_rank 1 --nproc_per_node ${gpu_num} --master_addr=192.168.1.1 --master_port=12345 \-../../../funasr/bin/train.py ${train_args}+../../../funasr/bin/train_ds.py ${train_args}```--nnodes indicates the total number of nodes participating in the training, --node_rank represents the ID of the current node, and --nproc_per_node specifies the number of processes running on each node (usually corresponds to the number of GPUs).@@ -321,6 +321,28 @@ jsonl2scp \++jsonl_file_in="../../../data/list/train.jsonl"```++#### Large-Scale Data Training+When dealing with large datasets (e.g., 50,000 hours or more), memory issues may arise, especially in multi-GPU experiments. To address this, split the JSONL files into slices, write them into a TXT file (one slice per line), and set `data_split_num`. For example:+```shell+train_data="/root/data/list/data.list"++funasr/bin/train_ds.py \+++train_data_set_list="${train_data}" \+++dataset_conf.data_split_num=256+```+**Details:**+- `data.list`: A plain text file listing the split JSONL files. For example, the content of `data.list` might be:+ ```bash+ data/list/train.0.jsonl+ data/list/train.1.jsonl+ ...+ ```+- `data_split_num`: Specifies the number of slice groups. For instance, if `data.list` contains 512 lines and `data_split_num=256`, the data will be divided into 256 groups, each containing 2 JSONL files. This ensures that only 2 JSONL files are loaded for training at a time, reducing memory usage during training. Note: Groups are created sequentially.++**Recommendation:**+If the dataset is extremely large and contains heterogeneous data types, perform **data balancing** during splitting to ensure uniformity across groups.+#### Training log##### log.txtdiff --git a/docs/tutorial/README_zh.md b/docs/tutorial/README_zh.mdindex 5bf15f0a4..6cbd9c38e 100644--- a/docs/tutorial/README_zh.md+++ b/docs/tutorial/README_zh.md@@ -213,7 +213,7 @@ bash finetune.sh### 详细参数介绍```shell-funasr/bin/train.py \+funasr/bin/train_ds.py \++model="${model_name_or_model_dir}" \++train_data_set_list="${train_data}" \++valid_data_set_list="${val_data}" \@@ -258,7 +258,7 @@ export CUDA_VISIBLE_DEVICES="0,1"gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')torchrun --nnodes 1 --nproc_per_node ${gpu_num} \-../../../funasr/bin/train.py ${train_args}+../../../funasr/bin/train_ds.py ${train_args}```--nnodes 表示参与的节点总数,--nproc_per_node 表示每个节点上运行的进程数@@ -270,7 +270,7 @@ export CUDA_VISIBLE_DEVICES="0,1"gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')torchrun --nnodes 2 --node_rank 0 --nproc_per_node ${gpu_num} --master_addr 192.168.1.1 --master_port 12345 \-../../../funasr/bin/train.py ${train_args}+../../../funasr/bin/train_ds.py ${train_args}```在从节点上(假设IP为192.168.1.2),你需要确保MASTER_ADDR和MASTER_PORT环境变量与主节点设置的一致,并运行同样的命令:```shell@@ -278,7 +278,7 @@ export CUDA_VISIBLE_DEVICES="0,1"gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')torchrun --nnodes 2 --node_rank 1 --nproc_per_node ${gpu_num} --master_addr 192.168.1.1 --master_port 12345 \-../../../funasr/bin/train.py ${train_args}+../../../funasr/bin/train_ds.py ${train_args}```--nnodes 表示参与的节点总数,--node_rank 表示当前节点id,--nproc_per_node 表示每个节点上运行的进程数(通常为gpu个数)@@ -331,6 +331,25 @@ jsonl2scp \++jsonl_file_in="../../../data/list/train.jsonl"```+#### 大数据训练+如果数据量很大,例如5万小时以上,这时候容易遇到内存不足的问题,特别是多gpu实验,这时候需要对jsonl文件进行切分成slice,然后写到txt里面,一行一个slice,然后设置`data_split_num`,例如:+```shell+train_data="/root/data/list/data.list"++funasr/bin/train_ds.py \+++train_data_set_list="${train_data}" \+++dataset_conf.data_split_num=256+```+其中:+`data.list`:为纯文本,内容是切割后的jsonl文件,例如,`data.list`的内容为:+```bash+data/list/train.0.jsonl+data/list/train.1.jsonl+...+```+`data_split_num`:表示切分slice分组个数,例如,data.list中共512行,data_split_num=256,表示分成256组,每组有2个jsonl文件,这样每次只load 2个jsonl数据进行训练,从而降低训练过程中内存使用。注意是按照顺序分组。+如果是,非常大,并且数据类型差异比较大,建议切分时候进行数据均衡。+#### 查看训练日志##### 查看实验log