Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
feat: implement Whisper forward() for fine-tuning support (fixes #2678)
游雁a8d23eecbb45
diff --git a/examples/industrial_data_pretraining/whisper/README.md b/examples/industrial_data_pretraining/whisper/README.mdnew file mode 100644index 000000000..3d3f6c207--- /dev/null+++ b/examples/industrial_data_pretraining/whisper/README.md@@ -0,0 +1,65 @@+# Whisper Fine-tuning with FunASR++Fine-tune OpenAI Whisper models on your own data using FunASR's training framework.++## Supported Models++- whisper-tiny / whisper-tiny.en+- whisper-base / whisper-base.en+- whisper-small / whisper-small.en+- whisper-medium / whisper-medium.en+- whisper-large-v1 / whisper-large-v2 / whisper-large-v3 / whisper-large-v3-turbo++## Data Preparation++Prepare data in JSONL format:++```json+{"key": "utt001", "source": "/path/to/audio1.wav", "target": "the transcription text"}+{"key": "utt002", "source": "/path/to/audio2.wav", "target": "another transcription"}+```++## Fine-tuning++```bash+bash finetune.sh+```++Or customize directly:++```python+from funasr import AutoModel++model = AutoModel(model="Whisper-large-v3", model_conf={"hub": "openai"})++# Training uses the forward() method which computes cross-entropy loss+# on (mel-spectrogram, token_ids) pairs+```++## Key Parameters++| Parameter | Default | Description |+|-----------|---------|-------------|+| model | Whisper-large-v3 | Model size |+| lr | 1e-5 | Learning rate (lower for larger models) |+| max_epoch | 10 | Training epochs |+| batch_size | 4 | Per-GPU batch size |+| warmup_steps | 500 | LR warmup |++## Tips++- For Chinese fine-tuning, use `whisper-large-v3` (best multilingual base)+- Freeze encoder for faster training: add `++train_conf.freeze_param="model.encoder"`+- Use smaller learning rates (1e-5 ~ 5e-6) to avoid catastrophic forgetting+- Recommended: 100+ hours of target-domain audio for meaningful improvement++## After Fine-tuning++```python+from funasr import AutoModel++# Load fine-tuned model+model = AutoModel(model="/path/to/exp/whisper_finetune")+result = model.generate(input="test.wav")+print(result[0]["text"])+```diff --git a/examples/industrial_data_pretraining/whisper/finetune.sh b/examples/industrial_data_pretraining/whisper/finetune.shnew file mode 100755index 000000000..3bbb3e3fd--- /dev/null+++ b/examples/industrial_data_pretraining/whisper/finetune.sh@@ -0,0 +1,31 @@+#!/bin/bash+# Whisper Fine-tuning with FunASR+#+# This script fine-tunes OpenAI Whisper models on custom data using FunASR's training framework.+# Supports: whisper-tiny, whisper-base, whisper-small, whisper-medium, whisper-large-v3+#+# Data format: JSONL with "audio" and "text" fields+# {"key": "utt1", "source": "/path/to/audio.wav", "target": "transcription text"}++export CUDA_VISIBLE_DEVICES=0,1++model_name="Whisper-large-v3"+train_data="data/train.jsonl"+val_data="data/val.jsonl"+output_dir="exp/whisper_finetune"++python -m funasr.bin.train \+ ++model="${model_name}" \+ ++model_conf.hub="openai" \+ ++train_data_set_list="${train_data}" \+ ++valid_data_set_list="${val_data}" \+ ++dataset_conf.batch_size=4 \+ ++dataset_conf.num_workers=4 \+ ++train_conf.output_dir="${output_dir}" \+ ++train_conf.max_epoch=10 \+ ++train_conf.lr=1e-5 \+ ++train_conf.warmup_steps=500 \+ ++optim="adam" \+ ++optim_conf.lr=1e-5 \+ ++scheduler="warmuplr" \+ ++scheduler_conf.warmup_steps=500diff --git a/funasr/models/whisper/model.py b/funasr/models/whisper/model.pyindex a8f183b89..15153884a 100644--- a/funasr/models/whisper/model.py+++ b/funasr/models/whisper/model.py@@ -65,9 +65,48 @@ class WhisperWarp(nn.Module):def forward(self,+ speech: torch.Tensor = None,+ speech_lengths: torch.Tensor = None,+ text: torch.Tensor = None,+ text_lengths: torch.Tensor = None,+ **kwargs,):- """Forward pass for training."""- pass+ """Forward pass for training. Computes cross-entropy loss.++ Args:+ speech: (B, T, D) mel-spectrogram features+ speech_lengths: (B,) lengths of each audio+ text: (B, U) token IDs (with SOT/EOT tokens)+ text_lengths: (B,) lengths of each text sequence+ Returns:+ dict with "loss" and optionally "stats"+ """+ if speech is None or text is None:+ raise ValueError("forward() requires speech and text for training")++ # Encoder+ audio_features = self.model.encoder(speech)++ # Decoder: shift text right for teacher forcing+ # text format: [SOT, lang, task, ..., tokens, EOT]+ decoder_input = text[:, :-1]+ decoder_target = text[:, 1:]++ # Decoder forward+ logits = self.model.decoder(decoder_input, audio_features)++ # Cross-entropy loss (ignore padding, token_id = -1 or pad)+ loss = F.cross_entropy(+ logits.reshape(-1, logits.size(-1)),+ decoder_target.reshape(-1),+ ignore_index=-100,+ )++ stats = {+ "loss": loss.detach().item(),+ "batch_size": speech.size(0),+ }+ return {"loss": loss, "stats": stats}def inference(self,