Skip to content

Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.

Add the dubbing engine: CosyVoice3 pack, worker, exact-length line placement, per-line reference...

Bowen Xuecd8f68d3d6d3
diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md
index 64906f6..647e47b 100644
--- a/THIRD_PARTY_NOTICES.md
+++ b/THIRD_PARTY_NOTICES.md
@@ -17,6 +17,7 @@ FreeVideo code uses [Apache-2.0](LICENSE). Its dependencies and models retain th
| [FreeToken](https://github.com/FlashML-org/FreeToken) prompt VLM operators, [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) optional weights | [Apache-2.0](freevideo_engine/licenses/FreeToken-Apache-2.0.txt) |
| [Qwen-Image-2.1](https://huggingface.co/Comfy-Org/Qwen-Image-2.1) Turbo image generation models, including Qwen3-VL-8B and VAE (optional download; [notice](freevideo_engine/image_gen/licenses/NOTICE.txt) installed with the models), © 2026 Hangzhou Tongyi Laboratory Technology Co., Ltd. All Rights Reserved. | [Qwen RESEARCH LICENSE AGREEMENT](https://huggingface.co/Comfy-Org/Qwen-Image-2.1/blob/main/LICENSE) |
| [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) 1.13.8 speech runtime, built by FreeVideo without text-to-speech, ALSA and PortAudio (modified build; `__init__.py` edited; source snapshot 11afbd00 kept by FreeVideo; optional download for automatic subtitles, [build note](freevideo_engine/licenses/sherpa-onnx-build-NOTE.txt)). Includes statically linked kaldi-native-fbank, kaldi-decoder, kaldifst, OpenFst, simple-sentencepiece (Apache-2.0), KissFFT (BSD-3-Clause), pybind11 (BSD-3-Clause), nlohmann/json (MIT), hclust-cpp and Darts-clone (BSD-2-Clause), ThreadPool (Zlib). The Windows build also contains msvcp140.dll and msvcp140_1.dll (Microsoft Visual C++ 2022 runtime 14.44.35211, © Microsoft Corporation), unmodified and redistributed as permitted by Microsoft for its redistributable files. | Apache-2.0 |
+| [CosyVoice](https://github.com/FunAudioLLM/CosyVoice) code adapted for dubbing and [Fun-CosyVoice3-0.5B-2512](https://modelscope.cn/models/FunAudioLLM/Fun-CosyVoice3-0.5B-2512) weights, © 2024 Alibaba Inc (optional download); includes MIT-licensed parts of Matcha-TTS, HiFi-GAN, x_transformers and Snake | [Apache-2.0](freevideo_engine/licenses/CosyVoice-Apache-2.0.txt); [MIT notices](freevideo_engine/licenses/CosyVoice-NOTICE.txt) |
| [ONNX Runtime](https://github.com/microsoft/onnxruntime) 1.28.2 (CPU), © Microsoft Corporation; its third-party notices are shipped inside the sherpa-onnx wheel | MIT |
| [Eigen](https://gitlab.com/libeigen/eigen) 5.0.1, compiled into sherpa-onnx and ONNX Runtime; source: https://gitlab.com/libeigen/eigen/-/tree/5.0.1 | MPL-2.0 |
| [Silero VAD](https://github.com/snakers4/silero-vad) v5.0 voice-activity model, © 2020-present Silero Team (optional download with subtitles, license file downloaded with the model; the onnx-community export is downloaded with the music model) | [MIT](freevideo_engine/licenses/Silero-VAD-MIT.txt) |
diff --git a/__init__.py b/__init__.py
index 3349199..b6214c5 100644
--- a/__init__.py
+++ b/__init__.py
@@ -31,6 +31,12 @@ async def comfy_entrypoint():
except Exception as error:
import logging
logging.warning('[FreeVideo] Image generation routes not registered: %s', error)
+ try:
+ from .freevideo_engine.dubbing.service import register as dubbing_routes
+ dubbing_routes()
+ except Exception as error:
+ import logging
+ logging.warning('[FreeVideo] Dubbing routes not registered: %s', error)
from .freevideo_engine.comfy_launcher_api import register as launcher_routes
launcher_routes()
from .freevideo_engine.comfy_updates import register as update_routes
diff --git a/constraints/unified.txt b/constraints/unified.txt
index 882a9d6..b281491 100644
--- a/constraints/unified.txt
+++ b/constraints/unified.txt
@@ -17,6 +17,7 @@ cuda-toolkit==13.0.3.0
cryptography==50.0.1
einops==0.8.2
filelock==3.32.3
+flatbuffers==25.12.19
fsspec==2026.7.0
h11==0.16.0
hf-xet==1.6.1a0
@@ -50,10 +51,12 @@ nvidia-nvjitlink==13.3.33
nvidia-nvshmem-cu13==3.4.5
nvidia-nvtx==13.0.85
omegaconf==2.3.1
+onnxruntime==1.31.1
packaging==26.3
peft==0.20.0
pillow==12.3.0
pip==26.2.1
+protobuf==7.36.2
psutil==7.2.2
pygments==2.21.0
pycparser==3.0
diff --git a/freevideo_engine/dubbing/__init__.py b/freevideo_engine/dubbing/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/freevideo_engine/dubbing/align.py b/freevideo_engine/dubbing/align.py
new file mode 100644
index 0000000..5c3ec0d
--- /dev/null
+++ b/freevideo_engine/dubbing/align.py
@@ -0,0 +1,238 @@
+"""Fit a synthesized line to a target duration, whatever backend made it.
+
+A backend exposes `synth(speed)` -> (mono float32 samples, sample rate); `speed`
+above 1 makes the line shorter. The first pass is natural speed, the second
+pass uses the measured ratio, and a third corrects what is still outside the
+tolerance. The voiced part (first to last loud 5 ms frame) is then closed to the wanted length by a
+pitch-preserving tempo change within `max_stretch`, and placed at the wanted onset in a file of exactly
+the target number of samples: silence fills what is short. Voice that cannot fit is never cut: the file
+is then longer than the target and reports `overrun` seconds (`fitted` false). A few milliseconds of the quiet
+edge around the voiced part are kept, so soft consonants and the decay of the last sound survive.
+"""
+from fractions import Fraction
+
+import numpy as np
+
+TOLERANCE = 0.02
+FIT_SECONDS = 0.04
+FADE = 0.003
+EDGE = 0.02
+MEASURE_FRAME = 0.005
+MAX_STRETCH = 1.25
+SPEED_RANGE = (0.6, 1.8)
+
+
+LEAD_KEEP = 0.05
+TRAIL_KEEP = 0.12
+SILENCE_LEVEL = 0.02
+SPAN_SHARE = 0.02 # a line starts and ends where the smoothed energy crosses this share of its peak
+NOISE_MARGIN = 4. # the voiced level is at least this many times the noise floor (10th percentile of the frames)
+NOISE_CAP = 0.1 # ... but never above this share of the loudest frame
+
+
+def seconds(samples, rate):
+ return len(samples) / rate
+
+
+def trim_silence(samples, rate, lead_keep=LEAD_KEEP, trail_keep=TRAIL_KEEP, level=SILENCE_LEVEL):
+ """Cut the quiet ends a model leaves around a line, keeping a short margin of each.
+
+ The language model sometimes opens with seconds of silence; measuring that as speech would make
+ every pass miss the target. Quiet is below `level` times the loudest 10 ms frame.
+ """
+ samples = np.asarray(samples, dtype=np.float32)
+ frame = max(1, rate // 100)
+ count = len(samples) // frame
+ if count < 2:
+ return samples
+ rms = np.sqrt((samples[:count * frame].reshape(count, frame) ** 2).mean(axis=1))
+ loud = np.nonzero(rms > level * rms.max())[0]
+ if not len(loud):
+ return samples
+ start = max(0, int(loud[0] * frame - lead_keep * rate))
+ end = min(len(samples), int((loud[-1] + 1) * frame + trail_keep * rate))
+ return samples[start:end]
+
+
+def stretch(samples, rate, ratio):
+ """Return `samples` played `ratio` times faster (ratio > 1 shortens), pitch kept."""
+ import av
+ graph = av.filter.Graph()
+ source = graph.add_abuffer(format='flt', sample_rate=rate, layout='mono', time_base=Fraction(1, rate))
+ tempo = graph.add('atempo', '%.6f' % ratio)
+ sink = graph.add('abuffersink')
+ source.link_to(tempo)
+ tempo.link_to(sink)
+ graph.configure()
+ frame = av.AudioFrame.from_ndarray(np.ascontiguousarray(samples, dtype=np.float32)[None, :],
+ format='flt', layout='mono')
+ frame.sample_rate, frame.pts = rate, 0
+ graph.push(frame)
+ graph.push(None)
+ parts = []
+ while True:
+ try:
+ parts.append(sink.pull().to_ndarray()[0])
+ except (av.error.EOFError, av.error.BlockingIOError):
+ break
+ return np.concatenate(parts).astype(np.float32) if parts else np.zeros(0, np.float32)
+
+
+def voiced_span(samples, rate, level=SILENCE_LEVEL):
+ """(first, end) sample of the voiced part: from the first to the last loud 5 ms frame, or None."""
+ samples = np.asarray(samples, dtype=np.float32)
+ frame = max(1, int(rate * MEASURE_FRAME))
+ count = len(samples) // frame
+ if count < 1:
+ return None
+ rms = np.sqrt((samples[:count * frame].reshape(count, frame) ** 2).mean(axis=1))
+ if not rms.max() > 0:
+ return None
+ # A recording with music or room noise comes back with that noise floor: the level rises above it
+ # (never past NOISE_CAP of the loudest frame), and a lone loud frame is not voice.
+ threshold = min(max(level * rms.max(), NOISE_MARGIN * np.percentile(rms, 10)), NOISE_CAP * rms.max())
+ above = rms > threshold
+ joined = above & (np.concatenate([above[1:], [False]]) | np.concatenate([[False], above[:-1]]))
+ loud = np.nonzero(joined if joined.any() else above)[0]
+ if not len(loud):
+ return None
+ return int(loud[0] * frame), int((loud[-1] + 1) * frame)
+
+
+def energy_span(samples, rate, share=SPAN_SHARE):
+ """(first, end) sample where the 5 ms smoothed energy stays above `share` of its peak: the definition
+ the timeline of an original line is taken with. None for silence."""
+ data = np.asarray(samples, dtype=np.float64)
+ window = max(1, int(rate * MEASURE_FRAME))
+ if len(data) < window:
+ return None
+ total = np.concatenate([[0.], np.cumsum(data * data)])
+ half = window // 2
+ centre = np.arange(len(data))
+ low, high = np.clip(centre - half, 0, len(data)), np.clip(centre - half + window, 0, len(data))
+ level = np.sqrt((total[high] - total[low]) / window)
+ if not level.max() > 0:
+ return None
+ loud = np.nonzero(level > share * level.max())[0]
+ return int(loud[0]), int(loud[-1]) + 1
+
+
+def place(voice, rate, total, onset, lead=0, trail=0):
+ """`voice` (`lead` quiet samples, the voiced part, `trail` quiet samples) laid into silence of
+ round(total*rate) samples so that the voiced part starts at `onset`; returns (samples, overrun).
+
+ `overrun` is the number of voiced samples past the end of the line. The file is then longer by that
+ much and nothing is cut; otherwise the quiet tail is trimmed to the exact length. Also returns the
+ voiced part's first and end sample in the result: the placement is known, so it is not measured again
+ (a line that fills its slot has no silence left, and the measure's noise floor would then be speech)."""
+ size = int(round(total * rate))
+ voiced = len(voice) - lead - trail
+ start = int(round(onset * rate)) - lead
+ take = np.array(voice, dtype=np.float32)
+ if start < 0:
+ take, start, lead = take[-start:], 0, lead + start
+ overrun = max(0, start + lead + voiced - size)
+ if overrun <= int(MEASURE_FRAME * rate): # under the resolution of the measurement
+ overrun = 0
+ take = take[:size - start]
+ length = max(size, start + len(take))
+ first = start + max(lead, 0)
+ last = min(first + voiced, start + len(take))
+ for width, edge in ((int(FADE * rate), 'in'), (int(FADE * rate), 'out')):
+ width = min(width, len(take))
+ if width > 1:
+ ramp = np.linspace(0, 1, width, dtype=np.float32)
+ if edge == 'in':
+ take[:width] *= ramp
+ else:
+ take[-width:] *= ramp[::-1]
+ out = np.zeros(length, np.float32)
+ out[start:start + len(take)] = take
+ return out, overrun, first, last
+
+
+def fit(synth, target, *, onset=0., offset=None, tolerance=TOLERANCE, max_stretch=MAX_STRETCH,
+ speed_range=SPEED_RANGE, check=lambda: None):
+ """Run the passes; return dict(samples, rate, seconds, onset, offset, error, onset_error,
+ offset_error, speed, passes, stretch,
+ fitted, warning, overrun, natural). `samples` has exactly round(target*rate) samples unless the voice overruns; the voiced part is
+ wanted from `onset` to `offset` (default: the whole line)."""
+ if not target > 0:
+ raise ValueError('Target duration must be positive')
+ offset = target if offset is None else offset
+ wanted = offset - onset
+ if not 0 <= onset < offset <= target + 1e-9:
+ raise ValueError('The wanted voiced part must lie inside the line')
+ low, high = speed_range
+ tried = []
+
+ def run(speed):
+ check()
+ samples, rate = synth(speed)
+ span = voiced_span(samples, rate)
+ if span is None:
+ raise ValueError('The dubbing backend returned no audio')
+ samples = np.asarray(samples, dtype=np.float32)
+ edge = int(EDGE * rate)
+ voice = samples[span[0]:span[1]]
+ # quiet samples around the voiced part, kept as they are (not stretched)
+ edges = (samples[max(0, span[0] - edge):span[0]], samples[span[1]:span[1] + edge])
+ tried.append((speed, voice, rate, seconds(voice, rate), edges))
+ return tried[-1]
+
+ def off(entry):
+ return abs(entry[3] - wanted) / wanted
+
+ speed = 1.0
+ entry = run(speed)
+ while off(entry) > tolerance and len(tried) < 3:
+ # the length scales with 1/speed, so the next speed is the measured ratio
+ speed = min(max(speed * entry[3] / wanted, low), high)
+ if speed == tried[-1][0]:
+ break
+ entry = run(speed)
+ speed, voice, rate, length, edges = min(tried, key=off)
+ padded_edges = edges
+
+ def compose(start, width):
+ """The voiced part closed to `width` seconds and placed from `start`; (out, overrun, first, last, ratio)."""
+ source, ratio, current, length_now = voice, 1.0, voice, length
+ for _ in range(2):
+ want = min(max(ratio * length_now / width, 1 / max_stretch), max_stretch)
+ if abs(want - ratio) < 0.001 and ratio != 1.0 or abs(length_now - width) <= 0.004 and ratio == 1.0:
+ break
+ ratio = want
+ current = stretch(source, rate, ratio)
+ length_now = seconds(current, rate)
+ padded = np.concatenate([padded_edges[0], current, padded_edges[1]])
+ out, overrun, first, last = place(padded, rate, target, start, lead=len(padded_edges[0]),
+ trail=len(padded_edges[1]))
+ return out, overrun, first, last, ratio
+
+ # The placed file is measured the way the originals' timeline is (smoothed energy above a share of its
+ # peak): the tempo change and the quiet edges spread a little energy past the voiced part, so the first
+ # placement is corrected by what is measured, twice at most.
+ start, width = onset, wanted
+ best = None
+ for attempt in range(3):
+ out, overrun, first, last, ratio = compose(start, width)
+ span = energy_span(out, rate)
+ first, last = (span[0], span[1]) if span else (first, last)
+ first, last = first / rate, last / rate
+ miss = max(abs(first - onset), abs(last - offset)) + (overrun > 0)
+ if best is None or miss < best[0]:
+ best = (miss, out, overrun, first, last, ratio)
+ if miss <= 0.004 or attempt == 2:
+ break
+ start = max(0., start + (onset - first))
+ width = max(0.25 * wanted, width + (wanted - (last - first)))
+ _, out, overrun, first, last, ratio = best
+ error = (last - first) - wanted
+ onset_error, offset_error = first - onset, last - offset
+ overrun = overrun / rate
+ fitted = abs(onset_error) <= FIT_SECONDS and abs(offset_error) <= FIT_SECONDS and not overrun
+ warning = None if fitted else 'too_long' if (overrun or offset_error > 0) else 'too_short'
+ return dict(samples=out, rate=rate, seconds=len(out) / rate, onset=first, offset=last, error=error,
+ onset_error=onset_error, offset_error=offset_error,
+ speed=speed, passes=len(tried), stretch=ratio, fitted=fitted, warning=warning, overrun=overrun,
+ natural=tried[0][3])
diff --git a/freevideo_engine/dubbing/backends.py b/freevideo_engine/dubbing/backends.py
new file mode 100644
index 0000000..3ceed52
--- /dev/null
+++ b/freevideo_engine/dubbing/backends.py
@@ -0,0 +1,63 @@
+"""Dubbing backends behind one interface, so a model can be replaced without touching the service.
+
+A backend is built from its model folder, says a line in the voice of one reference recording and
+frees its memory in `close`. `speed` above 1 makes the line shorter; `align.fit` uses it to reach a
+target duration.
+"""
+import importlib
+
+REGISTRY = {
+ 'cosyvoice3': 'freevideo_engine.dubbing.backends:CosyVoice3Backend',
+}
+DEFAULT = 'cosyvoice3'
+MAX_REFERENCE_SECONDS = 29.0
+
+
+class Backend:
+ name = ''
+ sample_rate = 24000
+ languages = ()
+ # Whether the reference needs a transcript of what it says
+ needs_reference_text = True
+ supports_instruction = False
+
+ def __init__(self, model_dir, device='cuda'):
+ raise NotImplementedError
+
+ def synth(self, text, reference, reference_text, speed=1.0, instruction=None):
+ """Return (mono float32 samples, sample rate)."""
+ raise NotImplementedError
+
+ def close(self):
+ pass
+
+
+class CosyVoice3Backend(Backend):
+ name = 'cosyvoice3'
+ sample_rate = 24000
+ languages = ('zh', 'en', 'ja', 'ko', 'de', 'es', 'fr', 'it', 'ru')
+ supports_instruction = True
+
+ def __init__(self, model_dir, device='cuda'):
+ from .cosyvoice3.build import CosyVoice3
+ self.model = CosyVoice3(model_dir, device=device)
+
+ def synth(self, text, reference, reference_text, speed=1.0, instruction=None):
+ if instruction:
+ samples = self.model.instruct(text, instruction, reference, speed=speed)
+ else:
+ samples = self.model.zero_shot(text, reference_text, reference, speed=speed)
+ return samples, self.sample_rate
+
+ def close(self):
+ self.model = None
+
+
+def names():
+ return sorted(REGISTRY)
+
+
+def create(name, model_dir, device='cuda'):
+ """Build the named backend; an unknown name raises KeyError."""
+ module, _, cls = REGISTRY[name].partition(':')
+ return getattr(importlib.import_module(module), cls)(model_dir, device=device)
diff --git a/freevideo_engine/dubbing/cosyvoice3/__init__.py b/freevideo_engine/dubbing/cosyvoice3/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/freevideo_engine/dubbing/cosyvoice3/build.py b/freevideo_engine/dubbing/cosyvoice3/build.py
new file mode 100644
index 0000000..6e4c6a2
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/build.py
@@ -0,0 +1,115 @@
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Build Fun-CosyVoice3-0.5B from its published weights without hyperpyyaml.
+
+The parameters below are those of the model's cosyvoice3.yaml.
+"""
+import os
+from functools import partial
+
+import torch
+from omegaconf import DictConfig
+
+from freevideo_engine.dubbing.cosyvoice3.cli.frontend import CosyVoiceFrontEnd
+from freevideo_engine.dubbing.cosyvoice3.cli.model import CosyVoice3Model
+from freevideo_engine.dubbing.cosyvoice3.flow.DiT.dit import DiT
+from freevideo_engine.dubbing.cosyvoice3.flow.flow import CausalMaskedDiffWithDiT
+from freevideo_engine.dubbing.cosyvoice3.flow.flow_matching import CausalConditionalCFM
+from freevideo_engine.dubbing.cosyvoice3.hifigan.f0_predictor import CausalConvRNNF0Predictor
+from freevideo_engine.dubbing.cosyvoice3.hifigan.generator import CausalHiFTGenerator
+from freevideo_engine.dubbing.cosyvoice3.llm.llm import CosyVoice3LM, Qwen2Encoder
+from freevideo_engine.dubbing.cosyvoice3.tokenizer.tokenizer import get_qwen_tokenizer
+from freevideo_engine.dubbing.cosyvoice3.transformer.upsample_encoder import PreLookaheadLayer
+from freevideo_engine.dubbing.cosyvoice3.utils.common import ras_sampling
+from freevideo_engine.dubbing.cosyvoice3.utils.mel import mel_spectrogram
+
+SAMPLE_RATE = 24000
+QWEN_DIR = 'CosyVoice-BlankEN'
+PROMPT_PREFIX = 'You are a helpful assistant.<|endofprompt|>'
+
+
+def build_networks(qwen_dir):
+ llm = CosyVoice3LM(
+ llm_input_size=896, llm_output_size=896, speech_token_size=6561,
+ llm=Qwen2Encoder(qwen_dir),
+ sampling=partial(ras_sampling, top_p=0.8, top_k=25, win_size=10, tau_r=0.1),
+ mix_ratio=[5, 15])
+ flow = CausalMaskedDiffWithDiT(
+ input_size=80, output_size=80, spk_embed_dim=192, output_type='mel', vocab_size=6561,
+ input_frame_rate=25, only_mask_loss=True, token_mel_ratio=2, pre_lookahead_len=3,
+ pre_lookahead_layer=PreLookaheadLayer(in_channels=80, channels=1024, pre_lookahead_len=3),
+ decoder=CausalConditionalCFM(
+ in_channels=240, n_spks=1, spk_emb_dim=80,
+ cfm_params=DictConfig({'sigma_min': 1e-06, 'solver': 'euler', 't_scheduler': 'cosine',
+ 'training_cfg_rate': 0.2, 'inference_cfg_rate': 0.7, 'reg_loss_type': 'l1'}),
+ estimator=DiT(dim=1024, depth=22, heads=16, dim_head=64, ff_mult=2, mel_dim=80, mu_dim=80,
+ spk_dim=80, out_channels=80, static_chunk_size=50, num_decoding_left_chunks=-1)))
+ hift = CausalHiFTGenerator(
+ in_channels=80, base_channels=512, nb_harmonics=8, sampling_rate=SAMPLE_RATE,
+ nsf_alpha=0.1, nsf_sigma=0.003, nsf_voiced_threshold=10,
+ upsample_rates=[8, 5, 3], upsample_kernel_sizes=[16, 11, 7],
+ istft_params={'n_fft': 16, 'hop_len': 4},
+ resblock_kernel_sizes=[3, 7, 11], resblock_dilation_sizes=[[1, 3, 5]] * 3,
+ source_resblock_kernel_sizes=[7, 7, 11], source_resblock_dilation_sizes=[[1, 3, 5]] * 3,
+ lrelu_slope=0.1, audio_limit=0.99, conv_pre_look_right=4,
+ f0_predictor=CausalConvRNNF0Predictor(num_class=1, in_channels=80, cond_channels=512))
+ return llm, flow, hift
+
+
+def _feat_extractor():
+ return partial(mel_spectrogram, n_fft=1920, num_mels=80, sampling_rate=SAMPLE_RATE, hop_size=480,
+ win_size=1920, fmin=0, fmax=None, center=False)
+
+
+class CosyVoice3:
+ """Zero-shot voice cloning with Fun-CosyVoice3-0.5B; every call returns one float32 numpy waveform."""
+
+ sample_rate = SAMPLE_RATE
+
+ def __init__(self, model_dir, device='cuda', fp16=False):
+ qwen_dir = os.path.join(model_dir, QWEN_DIR)
+ self.frontend = CosyVoiceFrontEnd(
+ lambda: get_qwen_tokenizer(token_path=qwen_dir, skip_special_tokens=True, version='cosyvoice3'),
+ _feat_extractor(),
+ os.path.join(model_dir, 'campplus.onnx'),
+ os.path.join(model_dir, 'speech_tokenizer_v3.onnx'),
+ device=device)
+ llm, flow, hift = build_networks(qwen_dir)
+ self.model = CosyVoice3Model(llm, flow, hift, device, fp16)
+ self.model.load(os.path.join(model_dir, 'llm.pt'), os.path.join(model_dir, 'flow.pt'),
+ os.path.join(model_dir, 'hift.pt'))
+
+ def _run(self, texts, make_input, speed):
+ import numpy as np
+ parts = []
+ for piece in self.frontend.text_normalize(texts, split=True, text_frontend=True):
+ inp = make_input(piece)
+ speech = self.model.tts(
+ text=inp['text'], flow_embedding=inp['flow_embedding'], llm_embedding=inp['llm_embedding'],
+ prompt_text=inp['prompt_text'],
+ llm_prompt_speech_token=inp.get('llm_prompt_speech_token', torch.zeros(1, 0, dtype=torch.int32)),
+ flow_prompt_speech_token=inp['flow_prompt_speech_token'],
+ prompt_speech_feat=inp['prompt_speech_feat'], speed=speed)
+ parts.append(speech.squeeze(0).float().cpu().numpy())
+ return np.concatenate(parts).astype('float32')
+
+ def zero_shot(self, text, prompt_text, prompt_wav, speed=1.0):
+ """Say `text` in the voice of `prompt_wav`, whose transcript is `prompt_text`."""
+ prompt = PROMPT_PREFIX + prompt_text
+ return self._run(text, lambda t: self.frontend.frontend_zero_shot(t, prompt, prompt_wav, SAMPLE_RATE), speed)
+
+ def instruct(self, text, instruction, prompt_wav, speed=1.0):
+ """Say `text` in the voice of `prompt_wav` in the manner `instruction` (a Chinese sentence) describes."""
+ prompt = f'You are a helpful assistant. {instruction}<|endofprompt|>'
+ return self._run(text, lambda t: self.frontend.frontend_instruct2(t, prompt, prompt_wav, SAMPLE_RATE), speed)
diff --git a/freevideo_engine/dubbing/cosyvoice3/cli/__init__.py b/freevideo_engine/dubbing/cosyvoice3/cli/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/freevideo_engine/dubbing/cosyvoice3/cli/frontend.py b/freevideo_engine/dubbing/cosyvoice3/cli/frontend.py
new file mode 100644
index 0000000..3781db4
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/cli/frontend.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Modified by FreeVideo: reduced to the zero-shot and instruct2 inputs; onnxruntime, wetext and
+# inflect are optional so the package imports without them.
+import logging
+import re
+from functools import partial
+from typing import Callable
+
+import numpy as np
+import torch
+import torchaudio.compliance.kaldi as kaldi
+
+from freevideo_engine.dubbing.cosyvoice3.utils.file_utils import load_wav
+from freevideo_engine.dubbing.cosyvoice3.utils.frontend_utils import (
+ contains_chinese, is_only_punctuation, remove_bracket, replace_blank, replace_corner_mark,
+ spell_out_number, split_paragraph)
+from freevideo_engine.dubbing.cosyvoice3.utils.mel import log_mel_spectrogram
+
+logger = logging.getLogger(__name__)
+
+
+class CosyVoiceFrontEnd:
+
+ def __init__(self,
+ get_tokenizer: Callable,
+ feat_extractor: Callable,
+ campplus_model: str,
+ speech_tokenizer_model: str,
+ device='cpu'):
+ import onnxruntime
+ self.tokenizer = get_tokenizer()
+ self.feat_extractor = feat_extractor
+ self.device = torch.device(device)
+ option = onnxruntime.SessionOptions()
+ option.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL
+ option.intra_op_num_threads = 1
+ # both small networks run on the CPU so they take no video memory next to the three generators
+ self.campplus_session = onnxruntime.InferenceSession(
+ campplus_model, sess_options=option, providers=['CPUExecutionProvider'])
+ self.speech_tokenizer_session = onnxruntime.InferenceSession(
+ speech_tokenizer_model, sess_options=option, providers=['CPUExecutionProvider'])
+ self.inflect_parser = None
+ try:
+ import inflect
+ self.inflect_parser = inflect.engine()
+ except Exception:
+ logger.info('inflect is not available, digits in English text are kept as written')
+ self.zh_tn_model = self.en_tn_model = None
+ try:
+ from wetext import Normalizer
+ self.zh_tn_model = Normalizer(remove_erhua=False)
+ self.en_tn_model = Normalizer()
+ except Exception:
+ logger.info('wetext is not available, text is not normalized')
+
+ def _extract_text_token(self, text):
+ text_token = self.tokenizer.encode(text)
+ text_token = torch.tensor([text_token], dtype=torch.int32).to(self.device)
+ text_token_len = torch.tensor([text_token.shape[1]], dtype=torch.int32).to(self.device)
+ return text_token, text_token_len
+
+ def _extract_speech_token(self, prompt_wav):
+ speech = load_wav(prompt_wav, 16000)
+ assert speech.shape[1] / 16000 <= 30, 'do not support extract speech token for audio longer than 30s'
+ feat = log_mel_spectrogram(speech, n_mels=128)
+ speech_token = self.speech_tokenizer_session.run(
+ None, {self.speech_tokenizer_session.get_inputs()[0].name: feat.detach().cpu().numpy(),
+ self.speech_tokenizer_session.get_inputs()[1].name: np.array([feat.shape[2]], dtype=np.int32)}
+ )[0].flatten().tolist()
+ speech_token = torch.tensor([speech_token], dtype=torch.int32).to(self.device)
+ speech_token_len = torch.tensor([speech_token.shape[1]], dtype=torch.int32).to(self.device)
+ return speech_token, speech_token_len
+
+ def _extract_spk_embedding(self, prompt_wav):
+ speech = load_wav(prompt_wav, 16000)
+ feat = kaldi.fbank(speech, num_mel_bins=80, dither=0, sample_frequency=16000)
+ feat = feat - feat.mean(dim=0, keepdim=True)
+ embedding = self.campplus_session.run(
+ None, {self.campplus_session.get_inputs()[0].name: feat.unsqueeze(dim=0).cpu().numpy()}
+ )[0].flatten().tolist()
+ return torch.tensor([embedding]).to(self.device)
+
+ def _extract_speech_feat(self, prompt_wav):
+ speech = load_wav(prompt_wav, 24000)
+ speech_feat = self.feat_extractor(speech).squeeze(dim=0).transpose(0, 1).to(self.device)
+ speech_feat = speech_feat.unsqueeze(dim=0)
+ speech_feat_len = torch.tensor([speech_feat.shape[1]], dtype=torch.int32).to(self.device)
+ return speech_feat, speech_feat_len
+
+ def text_normalize(self, text, split=True, text_frontend=True):
+ # NOTE skip the text frontend when a control symbol is in the text
+ if '<|' in text and '|>' in text:
+ text_frontend = False
+ if text_frontend is False or text == '':
+ return [text] if split is True else text
+ text = text.strip()
+ if contains_chinese(text):
+ if self.zh_tn_model is not None:
+ text = self.zh_tn_model.normalize(text)
+ text = text.replace("\n", "")
+ text = replace_blank(text)
+ text = replace_corner_mark(text)
+ text = text.replace(".", "。")
+ text = text.replace(" - ", ",")
+ text = remove_bracket(text)
+ text = re.sub(r'[,,、]+$', '。', text)
+ lang = 'zh'
+ else:
+ if self.en_tn_model is not None:
+ text = self.en_tn_model.normalize(text)
+ if self.inflect_parser is not None:
+ text = spell_out_number(text, self.inflect_parser)
+ lang = 'en'
+ texts = list(split_paragraph(text, self.tokenizer.encode, lang, token_max_n=80,
+ token_min_n=60, merge_len=20, comma_split=False))
+ texts = [i for i in texts if not is_only_punctuation(i)]
+ return texts if split is True else text
+
+ def frontend_zero_shot(self, tts_text, prompt_text, prompt_wav, resample_rate=24000):
+ tts_text_token, tts_text_token_len = self._extract_text_token(tts_text)
+ prompt_text_token, prompt_text_token_len = self._extract_text_token(prompt_text)
+ speech_feat, speech_feat_len = self._extract_speech_feat(prompt_wav)
+ speech_token, speech_token_len = self._extract_speech_token(prompt_wav)
+ if resample_rate == 24000:
+ # force speech_feat % speech_token = 2
+ token_len = min(int(speech_feat.shape[1] / 2), speech_token.shape[1])
+ speech_feat, speech_feat_len[:] = speech_feat[:, :2 * token_len], 2 * token_len
+ speech_token, speech_token_len[:] = speech_token[:, :token_len], token_len
+ embedding = self._extract_spk_embedding(prompt_wav)
+ return {'prompt_text': prompt_text_token, 'prompt_text_len': prompt_text_token_len,
+ 'llm_prompt_speech_token': speech_token, 'llm_prompt_speech_token_len': speech_token_len,
+ 'flow_prompt_speech_token': speech_token, 'flow_prompt_speech_token_len': speech_token_len,
+ 'prompt_speech_feat': speech_feat, 'prompt_speech_feat_len': speech_feat_len,
+ 'llm_embedding': embedding, 'flow_embedding': embedding,
+ 'text': tts_text_token, 'text_len': tts_text_token_len}
+
+ def frontend_instruct2(self, tts_text, instruct_text, prompt_wav, resample_rate=24000):
+ model_input = self.frontend_zero_shot(tts_text, instruct_text, prompt_wav, resample_rate)
+ del model_input['llm_prompt_speech_token']
+ del model_input['llm_prompt_speech_token_len']
+ return model_input
diff --git a/freevideo_engine/dubbing/cosyvoice3/cli/model.py b/freevideo_engine/dubbing/cosyvoice3/cli/model.py
new file mode 100644
index 0000000..b3ee887
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/cli/model.py
@@ -0,0 +1,88 @@
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu)
+# 2025 Alibaba Inc (authors: Xiang Lyu, Bofan Zhou)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Modified by FreeVideo: reduced to the non-streaming CosyVoice3 inference path
+# (no streaming, TensorRT, vLLM or JIT variants); the language model runs on the
+# calling thread so errors propagate.
+import torch
+from torch.nn import functional as F
+
+
+class CosyVoice3Model:
+
+ def __init__(self, llm: torch.nn.Module, flow: torch.nn.Module, hift: torch.nn.Module, device, fp16: bool = False):
+ self.device = torch.device(device)
+ self.llm = llm
+ self.flow = flow
+ self.hift = hift
+ self.fp16 = fp16
+ # FSQ silent and breath token
+ self.silent_tokens = [1, 2, 28, 29, 55, 248, 494, 2241, 2242, 2322, 2323]
+
+ def load(self, llm_model, flow_model, hift_model):
+ """Load the three state dicts onto the CPU first and then move each part to the device."""
+ for part, path in ((self.llm, llm_model), (self.flow, flow_model)):
+ part.load_state_dict(torch.load(path, map_location='cpu', weights_only=True), strict=True)
+ hift_state = torch.load(hift_model, map_location='cpu', weights_only=True)
+ self.hift.load_state_dict({k.replace('generator.', ''): v for k, v in hift_state.items()}, strict=True)
+ for part in (self.llm, self.flow, self.hift):
+ part.to(self.device).eval()
+
+ def speech_tokens(self, text, prompt_text, llm_prompt_speech_token, llm_embedding):
+ """Run the language model and return the generated speech tokens as a list."""
+ device = self.device
+ tokens, silent_run, max_silent_run = [], 0, 5
+ generator = self.llm.inference(
+ text=text.to(device),
+ text_len=torch.tensor([text.shape[1]], dtype=torch.int32).to(device),
+ prompt_text=prompt_text.to(device),
+ prompt_text_len=torch.tensor([prompt_text.shape[1]], dtype=torch.int32).to(device),
+ prompt_speech_token=llm_prompt_speech_token.to(device),
+ prompt_speech_token_len=torch.tensor([llm_prompt_speech_token.shape[1]], dtype=torch.int32).to(device),
+ embedding=llm_embedding.to(device),
+ uuid='freevideo')
+ for token in generator:
+ if token in self.silent_tokens:
+ silent_run += 1
+ if silent_run > max_silent_run:
+ continue
+ else:
+ silent_run = 0
+ tokens.append(token)
+ return tokens
+
+ def token2wav(self, token, prompt_token, prompt_feat, embedding, speed=1.0):
+ device = self.device
+ with torch.autocast(device_type=device.type, enabled=bool(self.fp16) and device.type == 'cuda'):
+ mel, _ = self.flow.inference(token=token.to(device, dtype=torch.int32),
+ token_len=torch.tensor([token.shape[1]], dtype=torch.int32).to(device),
+ prompt_token=prompt_token.to(device),
+ prompt_token_len=torch.tensor([prompt_token.shape[1]], dtype=torch.int32).to(device),
+ prompt_feat=prompt_feat.to(device),
+ prompt_feat_len=torch.tensor([prompt_feat.shape[1]], dtype=torch.int32).to(device),
+ embedding=embedding.to(device),
+ streaming=False,
+ finalize=True)
+ if speed != 1.0:
+ mel = F.interpolate(mel, size=int(mel.shape[2] / speed), mode='linear')
+ speech, _ = self.hift.inference(speech_feat=mel, finalize=True)
+ return speech
+
+ def tts(self, text, flow_embedding, llm_embedding, prompt_text, llm_prompt_speech_token,
+ flow_prompt_speech_token, prompt_speech_feat, speed=1.0):
+ tokens = self.speech_tokens(text, prompt_text, llm_prompt_speech_token, llm_embedding)
+ speech = self.token2wav(torch.tensor(tokens).unsqueeze(dim=0), flow_prompt_speech_token,
+ prompt_speech_feat, flow_embedding, speed=speed)
+ return speech.cpu()
diff --git a/freevideo_engine/dubbing/cosyvoice3/flow/DiT/__init__.py b/freevideo_engine/dubbing/cosyvoice3/flow/DiT/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/freevideo_engine/dubbing/cosyvoice3/flow/DiT/dit.py b/freevideo_engine/dubbing/cosyvoice3/flow/DiT/dit.py
new file mode 100644
index 0000000..e12fcee
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/flow/DiT/dit.py
@@ -0,0 +1,176 @@
+
+"""
+ein notation:
+b - batch
+n - sequence
+nt - text sequence
+nw - raw wave length
+d - dimension
+"""
+
+from __future__ import annotations
+
+import torch
+from torch import nn
+import torch.nn.functional as F
+from einops import repeat
+from freevideo_engine.dubbing.cosyvoice3.utils.mask import add_optional_chunk_mask
+from freevideo_engine.dubbing.cosyvoice3.flow.DiT.modules import (
+ RotaryEmbedding,
+ TimestepEmbedding,
+ ConvNeXtV2Block,
+ CausalConvPositionEmbedding,
+ DiTBlock,
+ AdaLayerNormZero_Final,
+ precompute_freqs_cis,
+ get_pos_embed_indices,
+)
+
+
+# Text embedding
+
+
+class TextEmbedding(nn.Module):
+ def __init__(self, text_num_embeds, text_dim, conv_layers=0, conv_mult=2):
+ super().__init__()
+ self.text_embed = nn.Embedding(text_num_embeds + 1, text_dim) # use 0 as filler token
+
+ if conv_layers > 0:
+ self.extra_modeling = True
+ self.precompute_max_pos = 4096 # ~44s of 24khz audio
+ self.register_buffer("freqs_cis", precompute_freqs_cis(text_dim, self.precompute_max_pos), persistent=False)
+ self.text_blocks = nn.Sequential(
+ *[ConvNeXtV2Block(text_dim, text_dim * conv_mult) for _ in range(conv_layers)]
+ )
+ else:
+ self.extra_modeling = False
+
+ def forward(self, text: int["b nt"], seq_len, drop_text=False): # noqa: F722
+ batch, text_len = text.shape[0], text.shape[1]
+ text = text + 1 # use 0 as filler token. preprocess of batch pad -1, see list_str_to_idx()
+ text = text[:, :seq_len] # curtail if character tokens are more than the mel spec tokens
+ text = F.pad(text, (0, seq_len - text_len), value=0)
+
+ if drop_text: # cfg for text
+ text = torch.zeros_like(text)
+
+ text = self.text_embed(text) # b n -> b n d
+
+ # possible extra modeling
+ if self.extra_modeling:
+ # sinus pos emb
+ batch_start = torch.zeros((batch,), dtype=torch.long)
+ pos_idx = get_pos_embed_indices(batch_start, seq_len, max_pos=self.precompute_max_pos)
+ text_pos_embed = self.freqs_cis[pos_idx]
+ text = text + text_pos_embed
+
+ # convnextv2 blocks
+ text = self.text_blocks(text)
+
+ return text
+
+
+# noised input audio and context mixing embedding
+
+
+class InputEmbedding(nn.Module):
+ def __init__(self, mel_dim, text_dim, out_dim, spk_dim=None):
+ super().__init__()
+ spk_dim = 0 if spk_dim is None else spk_dim
+ self.spk_dim = spk_dim
+ self.proj = nn.Linear(mel_dim * 2 + text_dim + spk_dim, out_dim)
+ self.conv_pos_embed = CausalConvPositionEmbedding(dim=out_dim)
+
+ def forward(
+ self,
+ x: float["b n d"],
+ cond: float["b n d"],
+ text_embed: float["b n d"],
+ spks: float["b d"],
+ ):
+ to_cat = [x, cond, text_embed]
+ if self.spk_dim > 0:
+ spks = repeat(spks, "b c -> b t c", t=x.shape[1])
+ to_cat.append(spks)
+
+ x = self.proj(torch.cat(to_cat, dim=-1))
+ x = self.conv_pos_embed(x) + x
+ return x
+
+
+# Transformer backbone using DiT blocks
+
+
+class DiT(nn.Module):
+ def __init__(
+ self,
+ *,
+ dim,
+ depth=8,
+ heads=8,
+ dim_head=64,
+ dropout=0.1,
+ ff_mult=4,
+ mel_dim=80,
+ mu_dim=None,
+ long_skip_connection=False,
+ spk_dim=None,
+ out_channels=None,
+ static_chunk_size=50,
+ num_decoding_left_chunks=2
+ ):
+ super().__init__()
+
+ self.time_embed = TimestepEmbedding(dim)
+ if mu_dim is None:
+ mu_dim = mel_dim
+ self.input_embed = InputEmbedding(mel_dim, mu_dim, dim, spk_dim)
+
+ self.rotary_embed = RotaryEmbedding(dim_head)
+
+ self.dim = dim
+ self.depth = depth
+
+ self.transformer_blocks = nn.ModuleList(
+ [DiTBlock(dim=dim, heads=heads, dim_head=dim_head, ff_mult=ff_mult, dropout=dropout) for _ in range(depth)]
+ )
+ self.long_skip_connection = nn.Linear(dim * 2, dim, bias=False) if long_skip_connection else None
+
+ self.norm_out = AdaLayerNormZero_Final(dim) # final modulation
+ self.proj_out = nn.Linear(dim, mel_dim)
+ self.out_channels = out_channels
+ self.static_chunk_size = static_chunk_size
+ self.num_decoding_left_chunks = num_decoding_left_chunks
+
+ def forward(self, x, mask, mu, t, spks=None, cond=None, streaming=False):
+ x = x.transpose(1, 2)
+ mu = mu.transpose(1, 2)
+ cond = cond.transpose(1, 2)
+ spks = spks.unsqueeze(dim=1)
+ batch, seq_len = x.shape[0], x.shape[1]
+ if t.ndim == 0:
+ t = t.repeat(batch)
+
+ # t: conditioning time, c: context (text + masked cond audio), x: noised input audio
+ t = self.time_embed(t)
+ x = self.input_embed(x, cond, mu, spks.squeeze(1))
+
+ rope = self.rotary_embed.forward_from_seq_len(seq_len)
+
+ if self.long_skip_connection is not None:
+ residual = x
+
+ if streaming is True:
+ attn_mask = add_optional_chunk_mask(x, mask.bool(), False, False, 0, self.static_chunk_size, -1).unsqueeze(dim=1)
+ else:
+ attn_mask = add_optional_chunk_mask(x, mask.bool(), False, False, 0, 0, -1).repeat(1, x.size(1), 1).unsqueeze(dim=1)
+
+ for block in self.transformer_blocks:
+ x = block(x, t, mask=attn_mask.bool(), rope=rope)
+
+ if self.long_skip_connection is not None:
+ x = self.long_skip_connection(torch.cat((x, residual), dim=-1))
+
+ x = self.norm_out(x, t)
+ output = self.proj_out(x).transpose(1, 2)
+ return output
diff --git a/freevideo_engine/dubbing/cosyvoice3/flow/DiT/modules.py b/freevideo_engine/dubbing/cosyvoice3/flow/DiT/modules.py
new file mode 100644
index 0000000..1b1cd86
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/flow/DiT/modules.py
@@ -0,0 +1,653 @@
+
+"""
+ein notation:
+b - batch
+n - sequence
+nt - text sequence
+nw - raw wave length
+d - dimension
+"""
+
+from __future__ import annotations
+from typing import Optional
+import math
+
+import torch
+from torch import nn
+import torch.nn.functional as F
+import torchaudio
+
+
+
+# raw wav to mel spec
+class MelSpec(nn.Module):
+ def __init__(
+ self,
+ filter_length=1024,
+ hop_length=256,
+ win_length=1024,
+ n_mel_channels=100,
+ target_sample_rate=24_000,
+ normalize=False,
+ power=1,
+ norm=None,
+ center=True,
+ ):
+ super().__init__()
+ self.n_mel_channels = n_mel_channels
+
+ self.mel_stft = torchaudio.transforms.MelSpectrogram(
+ sample_rate=target_sample_rate,
+ n_fft=filter_length,
+ win_length=win_length,
+ hop_length=hop_length,
+ n_mels=n_mel_channels,
+ power=power,
+ center=center,
+ normalized=normalize,
+ norm=norm,
+ )
+
+ self.register_buffer("dummy", torch.tensor(0), persistent=False)
+
+ def forward(self, inp):
+ if len(inp.shape) == 3:
+ inp = inp.squeeze(1) # 'b 1 nw -> b nw'
+
+ assert len(inp.shape) == 2
+
+ if self.dummy.device != inp.device:
+ self.to(inp.device)
+
+ mel = self.mel_stft(inp)
+ mel = mel.clamp(min=1e-5).log()
+ return mel
+
+
+# sinusoidal position embedding
+
+
+class SinusPositionEmbedding(nn.Module):
+ def __init__(self, dim):
+ super().__init__()
+ self.dim = dim
+
+ def forward(self, x, scale=1000):
+ device = x.device
+ half_dim = self.dim // 2
+ emb = math.log(10000) / (half_dim - 1)
+ emb = torch.exp(torch.arange(half_dim, device=device).float() * -emb)
+ emb = scale * x.unsqueeze(1) * emb.unsqueeze(0)
+ emb = torch.cat((emb.sin(), emb.cos()), dim=-1)
+ return emb
+
+
+# convolutional position embedding
+
+
+class ConvPositionEmbedding(nn.Module):
+ def __init__(self, dim, kernel_size=31, groups=16):
+ super().__init__()
+ assert kernel_size % 2 != 0
+ self.conv1d = nn.Sequential(
+ nn.Conv1d(dim, dim, kernel_size, groups=groups, padding=kernel_size // 2),
+ nn.Mish(),
+ nn.Conv1d(dim, dim, kernel_size, groups=groups, padding=kernel_size // 2),
+ nn.Mish(),
+ )
+
+ def forward(self, x: float["b n d"], mask: bool["b n"] | None = None): # noqa: F722
+ if mask is not None:
+ mask = mask[..., None]
+ x = x.masked_fill(~mask, 0.0)
+
+ x = x.permute(0, 2, 1)
+ x = self.conv1d(x)
+ out = x.permute(0, 2, 1)
+
+ if mask is not None:
+ out = out.masked_fill(~mask, 0.0)
+
+ return out
+
+
+class CausalConvPositionEmbedding(nn.Module):
+ def __init__(self, dim, kernel_size=31, groups=16):
+ super().__init__()
+ assert kernel_size % 2 != 0
+ self.kernel_size = kernel_size
+ self.conv1 = nn.Sequential(
+ nn.Conv1d(dim, dim, kernel_size, groups=groups, padding=0),
+ nn.Mish(),
+ )
+ self.conv2 = nn.Sequential(
+ nn.Conv1d(dim, dim, kernel_size, groups=groups, padding=0),
+ nn.Mish(),
+ )
+
+ def forward(self, x: float["b n d"], mask: bool["b n"] | None = None): # noqa: F722
+ if mask is not None:
+ mask = mask[..., None]
+ x = x.masked_fill(~mask, 0.0)
+
+ x = x.permute(0, 2, 1)
+ x = F.pad(x, (self.kernel_size - 1, 0, 0, 0))
+ x = self.conv1(x)
+ x = F.pad(x, (self.kernel_size - 1, 0, 0, 0))
+ x = self.conv2(x)
+ out = x.permute(0, 2, 1)
+
+ if mask is not None:
+ out = out.masked_fill(~mask, 0.0)
+
+ return out
+
+
+# rotary positional embedding, re-implemented from x_transformers (MIT) so the package is not a dependency
+
+
+class RotaryEmbedding(nn.Module):
+ def __init__(self, dim, base=10000):
+ super().__init__()
+ inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float() / dim))
+ self.register_buffer("inv_freq", inv_freq)
+
+ def forward_from_seq_len(self, seq_len):
+ t = torch.arange(seq_len, device=self.inv_freq.device)
+ with torch.autocast(t.device.type, enabled=False):
+ t = t.unsqueeze(0)
+ freqs = torch.einsum("b i , j -> b i j", t.type_as(self.inv_freq), self.inv_freq)
+ freqs = torch.stack((freqs, freqs), dim=-1)
+ freqs = freqs.reshape(*freqs.shape[:-2], -1)
+ return freqs, 1.0
+
+
+def rotate_half(x):
+ x = x.reshape(*x.shape[:-1], -1, 2)
+ x1, x2 = x.unbind(dim=-1)
+ x = torch.stack((-x2, x1), dim=-1)
+ return x.reshape(*x.shape[:-2], -1)
+
+
+def apply_rotary_pos_emb(t, freqs, scale=1):
+ rot_dim, seq_len, orig_dtype = freqs.shape[-1], t.shape[-2], t.dtype
+ with torch.autocast(t.device.type, enabled=False):
+ freqs = freqs[:, -seq_len:, :]
+ if t.ndim == 4 and freqs.ndim == 3:
+ freqs = freqs.unsqueeze(1)
+ t, t_unrotated = t[..., :rot_dim], t[..., rot_dim:]
+ t = (t * freqs.cos() * scale) + (rotate_half(t) * freqs.sin() * scale)
+ out = torch.cat((t, t_unrotated), dim=-1)
+ return out.type(orig_dtype)
+
+
+# rotary positional embedding related
+
+
+def precompute_freqs_cis(dim: int, end: int, theta: float = 10000.0, theta_rescale_factor=1.0):
+ # proposed by reddit user bloc97, to rescale rotary embeddings to longer sequence length without fine-tuning
+ # has some connection to NTK literature
+ # https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/
+ # https://github.com/lucidrains/rotary-embedding-torch/blob/main/rotary_embedding_torch/rotary_embedding_torch.py
+ theta *= theta_rescale_factor ** (dim / (dim - 2))
+ freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim))
+ t = torch.arange(end, device=freqs.device) # type: ignore
+ freqs = torch.outer(t, freqs).float() # type: ignore
+ freqs_cos = torch.cos(freqs) # real part
+ freqs_sin = torch.sin(freqs) # imaginary part
+ return torch.cat([freqs_cos, freqs_sin], dim=-1)
+
+
+def get_pos_embed_indices(start, length, max_pos, scale=1.0):
+ # length = length if isinstance(length, int) else length.max()
+ scale = scale * torch.ones_like(start, dtype=torch.float32) # in case scale is a scalar
+ pos = (
+ start.unsqueeze(1)
+ + (torch.arange(length, device=start.device, dtype=torch.float32).unsqueeze(0) * scale.unsqueeze(1)).long()
+ )
+ # avoid extra long error.
+ pos = torch.where(pos < max_pos, pos, max_pos - 1)
+ return pos
+
+
+# Global Response Normalization layer (Instance Normalization ?)
+
+
+class GRN(nn.Module):
+ def __init__(self, dim):
+ super().__init__()
+ self.gamma = nn.Parameter(torch.zeros(1, 1, dim))
+ self.beta = nn.Parameter(torch.zeros(1, 1, dim))
+
+ def forward(self, x):
+ Gx = torch.norm(x, p=2, dim=1, keepdim=True)
+ Nx = Gx / (Gx.mean(dim=-1, keepdim=True) + 1e-6)
+ return self.gamma * (x * Nx) + self.beta + x
+
+
+# ConvNeXt-V2 Block https://github.com/facebookresearch/ConvNeXt-V2/blob/main/models/convnextv2.py
+# ref: https://github.com/bfs18/e2_tts/blob/main/rfwave/modules.py#L108
+
+
+class ConvNeXtV2Block(nn.Module):
+ def __init__(
+ self,
+ dim: int,
+ intermediate_dim: int,
+ dilation: int = 1,
+ ):
+ super().__init__()
+ padding = (dilation * (7 - 1)) // 2
+ self.dwconv = nn.Conv1d(
+ dim, dim, kernel_size=7, padding=padding, groups=dim, dilation=dilation
+ ) # depthwise conv
+ self.norm = nn.LayerNorm(dim, eps=1e-6)
+ self.pwconv1 = nn.Linear(dim, intermediate_dim) # pointwise/1x1 convs, implemented with linear layers
+ self.act = nn.GELU()
+ self.grn = GRN(intermediate_dim)
+ self.pwconv2 = nn.Linear(intermediate_dim, dim)
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ residual = x
+ x = x.transpose(1, 2) # b n d -> b d n
+ x = self.dwconv(x)
+ x = x.transpose(1, 2) # b d n -> b n d
+ x = self.norm(x)
+ x = self.pwconv1(x)
+ x = self.act(x)
+ x = self.grn(x)
+ x = self.pwconv2(x)
+ return residual + x
+
+
+# AdaLayerNormZero
+# return with modulated x for attn input, and params for later mlp modulation
+
+
+class AdaLayerNormZero(nn.Module):
+ def __init__(self, dim):
+ super().__init__()
+
+ self.silu = nn.SiLU()
+ self.linear = nn.Linear(dim, dim * 6)
+
+ self.norm = nn.LayerNorm(dim, elementwise_affine=False, eps=1e-6)
+
+ def forward(self, x, emb=None):
+ emb = self.linear(self.silu(emb))
+ shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = torch.chunk(emb, 6, dim=1)
+
+ x = self.norm(x) * (1 + scale_msa[:, None]) + shift_msa[:, None]
+ return x, gate_msa, shift_mlp, scale_mlp, gate_mlp
+
+
+# AdaLayerNormZero for final layer
+# return only with modulated x for attn input, cuz no more mlp modulation
+
+
+class AdaLayerNormZero_Final(nn.Module):
+ def __init__(self, dim):
+ super().__init__()
+
+ self.silu = nn.SiLU()
+ self.linear = nn.Linear(dim, dim * 2)
+
+ self.norm = nn.LayerNorm(dim, elementwise_affine=False, eps=1e-6)
+
+ def forward(self, x, emb):
+ emb = self.linear(self.silu(emb))
+ scale, shift = torch.chunk(emb, 2, dim=1)
+
+ x = self.norm(x) * (1 + scale)[:, None, :] + shift[:, None, :]
+ return x
+
+
+# FeedForward
+
+
+class FeedForward(nn.Module):
+ def __init__(self, dim, dim_out=None, mult=4, dropout=0.0, approximate: str = "none"):
+ super().__init__()
+ inner_dim = int(dim * mult)
+ dim_out = dim_out if dim_out is not None else dim
+
+ activation = nn.GELU(approximate=approximate)
+ project_in = nn.Sequential(nn.Linear(dim, inner_dim), activation)
+ self.ff = nn.Sequential(project_in, nn.Dropout(dropout), nn.Linear(inner_dim, dim_out))
+
+ def forward(self, x):
+ return self.ff(x)
+
+
+# Attention with possible joint part
+# modified from diffusers/src/diffusers/models/attention_processor.py
+
+
+class Attention(nn.Module):
+ def __init__(
+ self,
+ processor: JointAttnProcessor | AttnProcessor,
+ dim: int,
+ heads: int = 8,
+ dim_head: int = 64,
+ dropout: float = 0.0,
+ context_dim: Optional[int] = None, # if not None -> joint attention
+ context_pre_only=None,
+ ):
+ super().__init__()
+
+ if not hasattr(F, "scaled_dot_product_attention"):
+ raise ImportError("Attention equires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.")
+
+ self.processor = processor
+
+ self.dim = dim
+ self.heads = heads
+ self.inner_dim = dim_head * heads
+ self.dropout = dropout
+
+ self.context_dim = context_dim
+ self.context_pre_only = context_pre_only
+
+ self.to_q = nn.Linear(dim, self.inner_dim)
+ self.to_k = nn.Linear(dim, self.inner_dim)
+ self.to_v = nn.Linear(dim, self.inner_dim)
+
+ if self.context_dim is not None:
+ self.to_k_c = nn.Linear(context_dim, self.inner_dim)
+ self.to_v_c = nn.Linear(context_dim, self.inner_dim)
+ if self.context_pre_only is not None:
+ self.to_q_c = nn.Linear(context_dim, self.inner_dim)
+
+ self.to_out = nn.ModuleList([])
+ self.to_out.append(nn.Linear(self.inner_dim, dim))
+ self.to_out.append(nn.Dropout(dropout))
+
+ if self.context_pre_only is not None and not self.context_pre_only:
+ self.to_out_c = nn.Linear(self.inner_dim, dim)
+
+ def forward(
+ self,
+ x: float["b n d"], # noised input x # noqa: F722
+ c: float["b n d"] = None, # context c # noqa: F722
+ mask: bool["b n"] | None = None, # noqa: F722
+ rope=None, # rotary position embedding for x
+ c_rope=None, # rotary position embedding for c
+ ) -> torch.Tensor:
+ if c is not None:
+ return self.processor(self, x, c=c, mask=mask, rope=rope, c_rope=c_rope)
+ else:
+ return self.processor(self, x, mask=mask, rope=rope)
+
+
+# Attention processor
+
+
+class AttnProcessor:
+ def __init__(self):
+ pass
+
+ def __call__(
+ self,
+ attn: Attention,
+ x: float["b n d"], # noised input x # noqa: F722
+ mask: bool["b n"] | None = None, # noqa: F722
+ rope=None, # rotary position embedding
+ ) -> torch.FloatTensor:
+ batch_size = x.shape[0]
+
+ # `sample` projections.
+ query = attn.to_q(x)
+ key = attn.to_k(x)
+ value = attn.to_v(x)
+
+ # apply rotary position embedding
+ if rope is not None:
+ freqs, xpos_scale = rope
+ q_xpos_scale, k_xpos_scale = (xpos_scale, xpos_scale**-1.0) if xpos_scale is not None else (1.0, 1.0)
+
+ query = apply_rotary_pos_emb(query, freqs, q_xpos_scale)
+ key = apply_rotary_pos_emb(key, freqs, k_xpos_scale)
+
+ # attention
+ inner_dim = key.shape[-1]
+ head_dim = inner_dim // attn.heads
+ query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
+ key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
+ value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
+
+ # mask. e.g. inference got a batch with different target durations, mask out the padding
+ if mask is not None:
+ attn_mask = mask
+ if attn_mask.dim() == 2:
+ attn_mask = attn_mask.unsqueeze(1).unsqueeze(1) # 'b n -> b 1 1 n'
+ attn_mask = attn_mask.expand(batch_size, attn.heads, query.shape[-2], key.shape[-2])
+ else:
+ attn_mask = None
+
+ x = F.scaled_dot_product_attention(query, key, value, attn_mask=attn_mask, dropout_p=0.0, is_causal=False)
+ x = x.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)
+ x = x.to(query.dtype)
+
+ # linear proj
+ x = attn.to_out[0](x)
+ # dropout
+ x = attn.to_out[1](x)
+
+ if mask is not None:
+ if mask.dim() == 2:
+ mask = mask.unsqueeze(-1)
+ else:
+ mask = mask[:, 0, -1].unsqueeze(-1)
+ x = x.masked_fill(~mask, 0.0)
+
+ return x
+
+
+# Joint Attention processor for MM-DiT
+# modified from diffusers/src/diffusers/models/attention_processor.py
+
+
+class JointAttnProcessor:
+ def __init__(self):
+ pass
+
+ def __call__(
+ self,
+ attn: Attention,
+ x: float["b n d"], # noised input x # noqa: F722
+ c: float["b nt d"] = None, # context c, here text # noqa: F722
+ mask: bool["b n"] | None = None, # noqa: F722
+ rope=None, # rotary position embedding for x
+ c_rope=None, # rotary position embedding for c
+ ) -> torch.FloatTensor:
+ residual = x
+
+ batch_size = c.shape[0]
+
+ # `sample` projections.
+ query = attn.to_q(x)
+ key = attn.to_k(x)
+ value = attn.to_v(x)
+
+ # `context` projections.
+ c_query = attn.to_q_c(c)
+ c_key = attn.to_k_c(c)
+ c_value = attn.to_v_c(c)
+
+ # apply rope for context and noised input independently
+ if rope is not None:
+ freqs, xpos_scale = rope
+ q_xpos_scale, k_xpos_scale = (xpos_scale, xpos_scale**-1.0) if xpos_scale is not None else (1.0, 1.0)
+ query = apply_rotary_pos_emb(query, freqs, q_xpos_scale)
+ key = apply_rotary_pos_emb(key, freqs, k_xpos_scale)
+ if c_rope is not None:
+ freqs, xpos_scale = c_rope
+ q_xpos_scale, k_xpos_scale = (xpos_scale, xpos_scale**-1.0) if xpos_scale is not None else (1.0, 1.0)
+ c_query = apply_rotary_pos_emb(c_query, freqs, q_xpos_scale)
+ c_key = apply_rotary_pos_emb(c_key, freqs, k_xpos_scale)
+
+ # attention
+ query = torch.cat([query, c_query], dim=1)
+ key = torch.cat([key, c_key], dim=1)
+ value = torch.cat([value, c_value], dim=1)
+
+ inner_dim = key.shape[-1]
+ head_dim = inner_dim // attn.heads
+ query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
+ key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
+ value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
+
+ # mask. e.g. inference got a batch with different target durations, mask out the padding
+ if mask is not None:
+ attn_mask = F.pad(mask, (0, c.shape[1]), value=True) # no mask for c (text)
+ attn_mask = attn_mask.unsqueeze(1).unsqueeze(1) # 'b n -> b 1 1 n'
+ attn_mask = attn_mask.expand(batch_size, attn.heads, query.shape[-2], key.shape[-2])
+ else:
+ attn_mask = None
+
+ x = F.scaled_dot_product_attention(query, key, value, attn_mask=attn_mask, dropout_p=0.0, is_causal=False)
+ x = x.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)
+ x = x.to(query.dtype)
+
+ # Split the attention outputs.
+ x, c = (
+ x[:, : residual.shape[1]],
+ x[:, residual.shape[1]:],
+ )
+
+ # linear proj
+ x = attn.to_out[0](x)
+ # dropout
+ x = attn.to_out[1](x)
+ if not attn.context_pre_only:
+ c = attn.to_out_c(c)
+
+ if mask is not None:
+ mask = mask.unsqueeze(-1)
+ x = x.masked_fill(~mask, 0.0)
+ # c = c.masked_fill(~mask, 0.) # no mask for c (text)
+
+ return x, c
+
+
+# DiT Block
+
+
+class DiTBlock(nn.Module):
+ def __init__(self, dim, heads, dim_head, ff_mult=4, dropout=0.1):
+ super().__init__()
+
+ self.attn_norm = AdaLayerNormZero(dim)
+ self.attn = Attention(
+ processor=AttnProcessor(),
+ dim=dim,
+ heads=heads,
+ dim_head=dim_head,
+ dropout=dropout,
+ )
+
+ self.ff_norm = nn.LayerNorm(dim, elementwise_affine=False, eps=1e-6)
+ self.ff = FeedForward(dim=dim, mult=ff_mult, dropout=dropout, approximate="tanh")
+
+ def forward(self, x, t, mask=None, rope=None): # x: noised input, t: time embedding
+ # pre-norm & modulation for attention input
+ norm, gate_msa, shift_mlp, scale_mlp, gate_mlp = self.attn_norm(x, emb=t)
+
+ # attention
+ attn_output = self.attn(x=norm, mask=mask, rope=rope)
+
+ # process attention output for input x
+ x = x + gate_msa.unsqueeze(1) * attn_output
+
+ ff_norm = self.ff_norm(x) * (1 + scale_mlp[:, None]) + shift_mlp[:, None]
+ ff_output = self.ff(ff_norm)
+ x = x + gate_mlp.unsqueeze(1) * ff_output
+
+ return x
+
+
+# MMDiT Block https://arxiv.org/abs/2403.03206
+
+
+class MMDiTBlock(nn.Module):
+ r"""
+ modified from diffusers/src/diffusers/models/attention.py
+
+ notes.
+ _c: context related. text, cond, etc. (left part in sd3 fig2.b)
+ _x: noised input related. (right part)
+ context_pre_only: last layer only do prenorm + modulation cuz no more ffn
+ """
+
+ def __init__(self, dim, heads, dim_head, ff_mult=4, dropout=0.1, context_pre_only=False):
+ super().__init__()
+
+ self.context_pre_only = context_pre_only
+
+ self.attn_norm_c = AdaLayerNormZero_Final(dim) if context_pre_only else AdaLayerNormZero(dim)
+ self.attn_norm_x = AdaLayerNormZero(dim)
+ self.attn = Attention(
+ processor=JointAttnProcessor(),
+ dim=dim,
+ heads=heads,
+ dim_head=dim_head,
+ dropout=dropout,
+ context_dim=dim,
+ context_pre_only=context_pre_only,
+ )
+
+ if not context_pre_only:
+ self.ff_norm_c = nn.LayerNorm(dim, elementwise_affine=False, eps=1e-6)
+ self.ff_c = FeedForward(dim=dim, mult=ff_mult, dropout=dropout, approximate="tanh")
+ else:
+ self.ff_norm_c = None
+ self.ff_c = None
+ self.ff_norm_x = nn.LayerNorm(dim, elementwise_affine=False, eps=1e-6)
+ self.ff_x = FeedForward(dim=dim, mult=ff_mult, dropout=dropout, approximate="tanh")
+
+ def forward(self, x, c, t, mask=None, rope=None, c_rope=None): # x: noised input, c: context, t: time embedding
+ # pre-norm & modulation for attention input
+ if self.context_pre_only:
+ norm_c = self.attn_norm_c(c, t)
+ else:
+ norm_c, c_gate_msa, c_shift_mlp, c_scale_mlp, c_gate_mlp = self.attn_norm_c(c, emb=t)
+ norm_x, x_gate_msa, x_shift_mlp, x_scale_mlp, x_gate_mlp = self.attn_norm_x(x, emb=t)
+
+ # attention
+ x_attn_output, c_attn_output = self.attn(x=norm_x, c=norm_c, mask=mask, rope=rope, c_rope=c_rope)
+
+ # process attention output for context c
+ if self.context_pre_only:
+ c = None
+ else: # if not last layer
+ c = c + c_gate_msa.unsqueeze(1) * c_attn_output
+
+ norm_c = self.ff_norm_c(c) * (1 + c_scale_mlp[:, None]) + c_shift_mlp[:, None]
+ c_ff_output = self.ff_c(norm_c)
+ c = c + c_gate_mlp.unsqueeze(1) * c_ff_output
+
+ # process attention output for input x
+ x = x + x_gate_msa.unsqueeze(1) * x_attn_output
+
+ norm_x = self.ff_norm_x(x) * (1 + x_scale_mlp[:, None]) + x_shift_mlp[:, None]
+ x_ff_output = self.ff_x(norm_x)
+ x = x + x_gate_mlp.unsqueeze(1) * x_ff_output
+
+ return c, x
+
+
+# time step conditioning embedding
+
+
+class TimestepEmbedding(nn.Module):
+ def __init__(self, dim, freq_embed_dim=256):
+ super().__init__()
+ self.time_embed = SinusPositionEmbedding(freq_embed_dim)
+ self.time_mlp = nn.Sequential(nn.Linear(freq_embed_dim, dim), nn.SiLU(), nn.Linear(dim, dim))
+
+ def forward(self, timestep: float["b"]): # noqa: F821
+ time_hidden = self.time_embed(timestep)
+ time_hidden = time_hidden.to(timestep.dtype)
+ time = self.time_mlp(time_hidden) # b d
+ return time
diff --git a/freevideo_engine/dubbing/cosyvoice3/flow/__init__.py b/freevideo_engine/dubbing/cosyvoice3/flow/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/freevideo_engine/dubbing/cosyvoice3/flow/flow.py b/freevideo_engine/dubbing/cosyvoice3/flow/flow.py
new file mode 100644
index 0000000..02f92f6
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/flow/flow.py
@@ -0,0 +1,103 @@
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Zhihao Du)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Modified by FreeVideo: reduced to the CosyVoice3 DiT flow inference path.
+from typing import Dict
+
+import torch
+import torch.nn as nn
+from torch.nn import functional as F
+from omegaconf import DictConfig
+
+from freevideo_engine.dubbing.cosyvoice3.utils.mask import make_pad_mask
+
+
+class CausalMaskedDiffWithDiT(torch.nn.Module):
+ def __init__(self,
+ input_size: int = 512,
+ output_size: int = 80,
+ spk_embed_dim: int = 192,
+ output_type: str = "mel",
+ vocab_size: int = 4096,
+ input_frame_rate: int = 50,
+ only_mask_loss: bool = True,
+ token_mel_ratio: int = 2,
+ pre_lookahead_len: int = 3,
+ pre_lookahead_layer: torch.nn.Module = None,
+ decoder: torch.nn.Module = None,
+ decoder_conf: Dict = {'in_channels': 240, 'out_channel': 80, 'spk_emb_dim': 80, 'n_spks': 1,
+ 'cfm_params': DictConfig({'sigma_min': 1e-06, 'solver': 'euler', 't_scheduler': 'cosine',
+ 'training_cfg_rate': 0.2, 'inference_cfg_rate': 0.7, 'reg_loss_type': 'l1'}),
+ 'decoder_params': {'channels': [256, 256], 'dropout': 0.0, 'attention_head_dim': 64,
+ 'n_blocks': 4, 'num_mid_blocks': 12, 'num_heads': 8, 'act_fn': 'gelu'}}):
+ super().__init__()
+ self.input_size = input_size
+ self.output_size = output_size
+ self.decoder_conf = decoder_conf
+ self.vocab_size = vocab_size
+ self.output_type = output_type
+ self.input_frame_rate = input_frame_rate
+ self.input_embedding = nn.Embedding(vocab_size, input_size)
+ self.spk_embed_affine_layer = torch.nn.Linear(spk_embed_dim, output_size)
+ self.pre_lookahead_len = pre_lookahead_len
+ self.pre_lookahead_layer = pre_lookahead_layer
+ self.decoder = decoder
+ self.only_mask_loss = only_mask_loss
+ self.token_mel_ratio = token_mel_ratio
+
+ @torch.inference_mode()
+ def inference(self,
+ token,
+ token_len,
+ prompt_token,
+ prompt_token_len,
+ prompt_feat,
+ prompt_feat_len,
+ embedding,
+ streaming,
+ finalize):
+ assert token.shape[0] == 1
+ # xvec projection
+ embedding = F.normalize(embedding, dim=1)
+ embedding = self.spk_embed_affine_layer(embedding)
+
+ # concat text and prompt_text
+ token, token_len = torch.concat([prompt_token, token], dim=1), prompt_token_len + token_len
+ mask = (~make_pad_mask(token_len)).unsqueeze(-1).to(embedding)
+ token = self.input_embedding(torch.clamp(token, min=0)) * mask
+
+ # text encode
+ if finalize is True:
+ h = self.pre_lookahead_layer(token)
+ else:
+ h = self.pre_lookahead_layer(token[:, :-self.pre_lookahead_len], context=token[:, -self.pre_lookahead_len:])
+ h = h.repeat_interleave(self.token_mel_ratio, dim=1)
+ mel_len1, mel_len2 = prompt_feat.shape[1], h.shape[1] - prompt_feat.shape[1]
+
+ # get conditions
+ conds = torch.zeros([1, mel_len1 + mel_len2, self.output_size], device=token.device).to(h.dtype)
+ conds[:, :mel_len1] = prompt_feat
+ conds = conds.transpose(1, 2)
+
+ mask = (~make_pad_mask(torch.tensor([mel_len1 + mel_len2]))).to(h)
+ feat, _ = self.decoder(
+ mu=h.transpose(1, 2).contiguous(),
+ mask=mask.unsqueeze(1),
+ spks=embedding,
+ cond=conds,
+ n_timesteps=10,
+ streaming=streaming
+ )
+ feat = feat[:, :, mel_len1:]
+ assert feat.shape[2] == mel_len2
+ return feat.float(), None
diff --git a/freevideo_engine/dubbing/cosyvoice3/flow/flow_matching.py b/freevideo_engine/dubbing/cosyvoice3/flow/flow_matching.py
new file mode 100644
index 0000000..eab641d
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/flow/flow_matching.py
@@ -0,0 +1,238 @@
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Zhihao Du)
+# 2025 Alibaba Inc (authors: Xiang Lyu, Bofan Zhou)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import torch
+import torch.nn.functional as F
+from freevideo_engine.dubbing.cosyvoice3.utils.common import set_all_random_seed
+
+
+class BASECFM(torch.nn.Module):
+ # inlined from Matcha-TTS (MIT): only the attributes the subclass uses
+ def __init__(self, n_feats, cfm_params, n_spks=1, spk_emb_dim=128):
+ super().__init__()
+ self.n_feats = n_feats
+ self.n_spks = n_spks
+ self.spk_emb_dim = spk_emb_dim
+ self.solver = cfm_params.solver
+ self.sigma_min = getattr(cfm_params, 'sigma_min', 1e-4)
+ self.estimator = None
+
+
+class ConditionalCFM(BASECFM):
+ def __init__(self, in_channels, cfm_params, n_spks=1, spk_emb_dim=64, estimator: torch.nn.Module = None):
+ super().__init__(
+ n_feats=in_channels,
+ cfm_params=cfm_params,
+ n_spks=n_spks,
+ spk_emb_dim=spk_emb_dim,
+ )
+ self.t_scheduler = cfm_params.t_scheduler
+ self.training_cfg_rate = cfm_params.training_cfg_rate
+ self.inference_cfg_rate = cfm_params.inference_cfg_rate
+ in_channels = in_channels + (spk_emb_dim if n_spks > 0 else 0)
+ # Just change the architecture of the estimator here
+ self.estimator = estimator
+
+ @torch.inference_mode()
+ def forward(self, mu, mask, n_timesteps, temperature=1.0, spks=None, cond=None, prompt_len=0, cache=torch.zeros(1, 80, 0, 2)):
+ """Forward diffusion
+
+ Args:
+ mu (torch.Tensor): output of encoder
+ shape: (batch_size, n_feats, mel_timesteps)
+ mask (torch.Tensor): output_mask
+ shape: (batch_size, 1, mel_timesteps)
+ n_timesteps (int): number of diffusion steps
+ temperature (float, optional): temperature for scaling noise. Defaults to 1.0.
+ spks (torch.Tensor, optional): speaker ids. Defaults to None.
+ shape: (batch_size, spk_emb_dim)
+ cond: Not used but kept for future purposes
+
+ Returns:
+ sample: generated mel-spectrogram
+ shape: (batch_size, n_feats, mel_timesteps)
+ """
+
+ z = torch.randn_like(mu).to(mu.device).to(mu.dtype) * temperature
+ cache_size = cache.shape[2]
+ # fix prompt and overlap part mu and z
+ if cache_size != 0:
+ z[:, :, :cache_size] = cache[:, :, :, 0]
+ mu[:, :, :cache_size] = cache[:, :, :, 1]
+ z_cache = torch.concat([z[:, :, :prompt_len], z[:, :, -34:]], dim=2)
+ mu_cache = torch.concat([mu[:, :, :prompt_len], mu[:, :, -34:]], dim=2)
+ cache = torch.stack([z_cache, mu_cache], dim=-1)
+
+ t_span = torch.linspace(0, 1, n_timesteps + 1, device=mu.device, dtype=mu.dtype)
+ if self.t_scheduler == 'cosine':
+ t_span = 1 - torch.cos(t_span * 0.5 * torch.pi)
+ return self.solve_euler(z, t_span=t_span, mu=mu, mask=mask, spks=spks, cond=cond), cache
+
+ def solve_euler(self, x, t_span, mu, mask, spks, cond, streaming=False):
+ """
+ Fixed euler solver for ODEs.
+ Args:
+ x (torch.Tensor): random noise
+ t_span (torch.Tensor): n_timesteps interpolated
+ shape: (n_timesteps + 1,)
+ mu (torch.Tensor): output of encoder
+ shape: (batch_size, n_feats, mel_timesteps)
+ mask (torch.Tensor): output_mask
+ shape: (batch_size, 1, mel_timesteps)
+ spks (torch.Tensor, optional): speaker ids. Defaults to None.
+ shape: (batch_size, spk_emb_dim)
+ cond: Not used but kept for future purposes
+ """
+ t, _, dt = t_span[0], t_span[-1], t_span[1] - t_span[0]
+ t = t.unsqueeze(dim=0)
+
+ # I am storing this because I can later plot it by putting a debugger here and saving it to a file
+ # Or in future might add like a return_all_steps flag
+ sol = []
+
+ # Do not use concat, it may cause memory format changed and trt infer with wrong results!
+ # NOTE when flow run in amp mode, x.dtype is float32, which cause nan in trt fp16 inference, so set dtype=spks.dtype
+ x_in = torch.zeros([2, 80, x.size(2)], device=x.device, dtype=spks.dtype)
+ mask_in = torch.zeros([2, 1, x.size(2)], device=x.device, dtype=spks.dtype)
+ mu_in = torch.zeros([2, 80, x.size(2)], device=x.device, dtype=spks.dtype)
+ t_in = torch.zeros([2], device=x.device, dtype=spks.dtype)
+ spks_in = torch.zeros([2, 80], device=x.device, dtype=spks.dtype)
+ cond_in = torch.zeros([2, 80, x.size(2)], device=x.device, dtype=spks.dtype)
+ for step in range(1, len(t_span)):
+ # Classifier-Free Guidance inference introduced in VoiceBox
+ x_in[:] = x
+ mask_in[:] = mask
+ mu_in[0] = mu
+ t_in[:] = t.unsqueeze(0)
+ spks_in[0] = spks
+ cond_in[0] = cond
+ dphi_dt = self.forward_estimator(
+ x_in, mask_in,
+ mu_in, t_in,
+ spks_in,
+ cond_in,
+ streaming
+ )
+ dphi_dt, cfg_dphi_dt = torch.split(dphi_dt, [x.size(0), x.size(0)], dim=0)
+ dphi_dt = ((1.0 + self.inference_cfg_rate) * dphi_dt - self.inference_cfg_rate * cfg_dphi_dt)
+ x = x + dt * dphi_dt
+ t = t + dt
+ sol.append(x)
+ if step < len(t_span) - 1:
+ dt = t_span[step + 1] - t
+
+ return sol[-1].float()
+
+ def forward_estimator(self, x, mask, mu, t, spks, cond, streaming=False):
+ if isinstance(self.estimator, torch.nn.Module):
+ return self.estimator(x, mask, mu, t, spks, cond, streaming=streaming)
+ else:
+ [estimator, stream], trt_engine = self.estimator.acquire_estimator()
+ # NOTE need to synchronize when switching stream
+ torch.cuda.current_stream().synchronize()
+ with stream:
+ estimator.set_input_shape('x', (2, 80, x.size(2)))
+ estimator.set_input_shape('mask', (2, 1, x.size(2)))
+ estimator.set_input_shape('mu', (2, 80, x.size(2)))
+ estimator.set_input_shape('t', (2,))
+ estimator.set_input_shape('spks', (2, 80))
+ estimator.set_input_shape('cond', (2, 80, x.size(2)))
+ data_ptrs = [x.contiguous().data_ptr(),
+ mask.contiguous().data_ptr(),
+ mu.contiguous().data_ptr(),
+ t.contiguous().data_ptr(),
+ spks.contiguous().data_ptr(),
+ cond.contiguous().data_ptr(),
+ x.data_ptr()]
+ for i, j in enumerate(data_ptrs):
+ estimator.set_tensor_address(trt_engine.get_tensor_name(i), j)
+ # run trt engine
+ assert estimator.execute_async_v3(torch.cuda.current_stream().cuda_stream) is True
+ torch.cuda.current_stream().synchronize()
+ self.estimator.release_estimator(estimator, stream)
+ return x
+
+ def compute_loss(self, x1, mask, mu, spks=None, cond=None, streaming=False):
+ """Computes diffusion loss
+
+ Args:
+ x1 (torch.Tensor): Target
+ shape: (batch_size, n_feats, mel_timesteps)
+ mask (torch.Tensor): target mask
+ shape: (batch_size, 1, mel_timesteps)
+ mu (torch.Tensor): output of encoder
+ shape: (batch_size, n_feats, mel_timesteps)
+ spks (torch.Tensor, optional): speaker embedding. Defaults to None.
+ shape: (batch_size, spk_emb_dim)
+
+ Returns:
+ loss: conditional flow matching loss
+ y: conditional flow
+ shape: (batch_size, n_feats, mel_timesteps)
+ """
+ b, _, t = mu.shape
+
+ # random timestep
+ t = torch.rand([b, 1, 1], device=mu.device, dtype=mu.dtype)
+
+ # sample noise p(x_0)
+ z = torch.randn_like(x1)
+
+ y = (1 - (1 - self.sigma_min) * t) * z + t * x1
+ u = x1 - (1 - self.sigma_min) * z
+
+ # during training, we randomly drop condition to trade off mode coverage and sample fidelity
+ if self.training_cfg_rate > 0:
+ cfg_mask = torch.rand(b, device=x1.device) > self.training_cfg_rate
+ mu = mu * cfg_mask.view(-1, 1, 1)
+ spks = spks * cfg_mask.view(-1, 1)
+ cond = cond * cfg_mask.view(-1, 1, 1)
+
+ pred = self.estimator(y, mask, mu, t.squeeze(), spks, cond, streaming=streaming)
+ loss = F.mse_loss(pred * mask, u * mask, reduction="sum") / (torch.sum(mask) * u.shape[1])
+ return loss, y
+
+
+class CausalConditionalCFM(ConditionalCFM):
+ def __init__(self, in_channels, cfm_params, n_spks=1, spk_emb_dim=64, estimator: torch.nn.Module = None):
+ super().__init__(in_channels, cfm_params, n_spks, spk_emb_dim, estimator)
+ set_all_random_seed(0)
+ self.rand_noise = torch.randn([1, 80, 50 * 300])
+
+ @torch.inference_mode()
+ def forward(self, mu, mask, n_timesteps, temperature=1.0, spks=None, cond=None, streaming=False):
+ """Forward diffusion
+
+ Args:
+ mu (torch.Tensor): output of encoder
+ shape: (batch_size, n_feats, mel_timesteps)
+ mask (torch.Tensor): output_mask
+ shape: (batch_size, 1, mel_timesteps)
+ n_timesteps (int): number of diffusion steps
+ temperature (float, optional): temperature for scaling noise. Defaults to 1.0.
+ spks (torch.Tensor, optional): speaker ids. Defaults to None.
+ shape: (batch_size, spk_emb_dim)
+ cond: Not used but kept for future purposes
+
+ Returns:
+ sample: generated mel-spectrogram
+ shape: (batch_size, n_feats, mel_timesteps)
+ """
+
+ z = self.rand_noise[:, :, :mu.size(2)].to(mu.device).to(mu.dtype) * temperature
+ # fix prompt and overlap part mu and z
+ t_span = torch.linspace(0, 1, n_timesteps + 1, device=mu.device, dtype=mu.dtype)
+ if self.t_scheduler == 'cosine':
+ t_span = 1 - torch.cos(t_span * 0.5 * torch.pi)
+ return self.solve_euler(z, t_span=t_span, mu=mu, mask=mask, spks=spks, cond=cond, streaming=streaming), None
diff --git a/freevideo_engine/dubbing/cosyvoice3/hifigan/__init__.py b/freevideo_engine/dubbing/cosyvoice3/hifigan/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/freevideo_engine/dubbing/cosyvoice3/hifigan/f0_predictor.py b/freevideo_engine/dubbing/cosyvoice3/hifigan/f0_predictor.py
new file mode 100644
index 0000000..25abfde
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/hifigan/f0_predictor.py
@@ -0,0 +1,103 @@
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Kai Hu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import torch
+import torch.nn as nn
+try:
+ from torch.nn.utils.parametrizations import weight_norm
+except ImportError:
+ from torch.nn.utils import weight_norm
+from freevideo_engine.dubbing.cosyvoice3.transformer.convolution import CausalConv1d
+
+
+class ConvRNNF0Predictor(nn.Module):
+ def __init__(self,
+ num_class: int = 1,
+ in_channels: int = 80,
+ cond_channels: int = 512
+ ):
+ super().__init__()
+
+ self.num_class = num_class
+ self.condnet = nn.Sequential(
+ weight_norm(
+ nn.Conv1d(in_channels, cond_channels, kernel_size=3, padding=1)
+ ),
+ nn.ELU(),
+ weight_norm(
+ nn.Conv1d(cond_channels, cond_channels, kernel_size=3, padding=1)
+ ),
+ nn.ELU(),
+ weight_norm(
+ nn.Conv1d(cond_channels, cond_channels, kernel_size=3, padding=1)
+ ),
+ nn.ELU(),
+ weight_norm(
+ nn.Conv1d(cond_channels, cond_channels, kernel_size=3, padding=1)
+ ),
+ nn.ELU(),
+ weight_norm(
+ nn.Conv1d(cond_channels, cond_channels, kernel_size=3, padding=1)
+ ),
+ nn.ELU(),
+ )
+ self.classifier = nn.Linear(in_features=cond_channels, out_features=self.num_class)
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ x = self.condnet(x)
+ x = x.transpose(1, 2)
+ return torch.abs(self.classifier(x).squeeze(-1))
+
+
+class CausalConvRNNF0Predictor(nn.Module):
+ def __init__(self,
+ num_class: int = 1,
+ in_channels: int = 80,
+ cond_channels: int = 512
+ ):
+ super().__init__()
+
+ self.num_class = num_class
+ self.condnet = nn.Sequential(
+ weight_norm(
+ CausalConv1d(in_channels, cond_channels, kernel_size=4, causal_type='right')
+ ),
+ nn.ELU(),
+ weight_norm(
+ CausalConv1d(cond_channels, cond_channels, kernel_size=3, causal_type='left')
+ ),
+ nn.ELU(),
+ weight_norm(
+ CausalConv1d(cond_channels, cond_channels, kernel_size=3, causal_type='left')
+ ),
+ nn.ELU(),
+ weight_norm(
+ CausalConv1d(cond_channels, cond_channels, kernel_size=3, causal_type='left')
+ ),
+ nn.ELU(),
+ weight_norm(
+ CausalConv1d(cond_channels, cond_channels, kernel_size=3, causal_type='left')
+ ),
+ nn.ELU(),
+ )
+ self.classifier = nn.Linear(in_features=cond_channels, out_features=self.num_class)
+
+ def forward(self, x: torch.Tensor, finalize: bool = True) -> torch.Tensor:
+ if finalize is True:
+ x = self.condnet[0](x)
+ else:
+ x = self.condnet[0](x[:, :, :-self.condnet[0].causal_padding], x[:, :, -self.condnet[0].causal_padding:])
+ for i in range(1, len(self.condnet)):
+ x = self.condnet[i](x)
+ x = x.transpose(1, 2)
+ return torch.abs(self.classifier(x).squeeze(-1))
diff --git a/freevideo_engine/dubbing/cosyvoice3/hifigan/generator.py b/freevideo_engine/dubbing/cosyvoice3/hifigan/generator.py
new file mode 100644
index 0000000..3766c84
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/hifigan/generator.py
@@ -0,0 +1,726 @@
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Kai Hu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""HIFI-GAN"""
+
+from typing import Dict, Optional, List
+import numpy as np
+from scipy.signal import get_window
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.nn import Conv1d
+from torch.nn import ConvTranspose1d
+from torch.nn.utils import remove_weight_norm
+try:
+ from torch.nn.utils.parametrizations import weight_norm
+except ImportError:
+ from torch.nn.utils import weight_norm
+from torch.distributions.uniform import Uniform
+from freevideo_engine.dubbing.cosyvoice3.transformer.convolution import CausalConv1d, CausalConv1dDownSample, CausalConv1dUpsample
+from freevideo_engine.dubbing.cosyvoice3.transformer.activation import Snake
+from freevideo_engine.dubbing.cosyvoice3.utils.common import get_padding
+from freevideo_engine.dubbing.cosyvoice3.utils.common import init_weights
+
+
+"""hifigan based generator implementation.
+
+This code is modified from https://github.com/jik876/hifi-gan
+ ,https://github.com/kan-bayashi/ParallelWaveGAN and
+ https://github.com/NVIDIA/BigVGAN
+
+"""
+
+
+class ResBlock(torch.nn.Module):
+ """Residual block module in HiFiGAN/BigVGAN."""
+ def __init__(
+ self,
+ channels: int = 512,
+ kernel_size: int = 3,
+ dilations: List[int] = [1, 3, 5],
+ causal: bool = False,
+ ):
+ super(ResBlock, self).__init__()
+ self.causal = causal
+ self.convs1 = nn.ModuleList()
+ self.convs2 = nn.ModuleList()
+
+ for dilation in dilations:
+ self.convs1.append(
+ weight_norm(
+ Conv1d(
+ channels,
+ channels,
+ kernel_size,
+ 1,
+ dilation=dilation,
+ padding=get_padding(kernel_size, dilation)) if causal is False else
+ CausalConv1d(
+ channels,
+ channels,
+ kernel_size,
+ 1,
+ dilation=dilation,
+ causal_type='left'
+ )
+ )
+ )
+ self.convs2.append(
+ weight_norm(
+ Conv1d(
+ channels,
+ channels,
+ kernel_size,
+ 1,
+ dilation=1,
+ padding=get_padding(kernel_size, 1)) if causal is False else
+ CausalConv1d(
+ channels,
+ channels,
+ kernel_size,
+ 1,
+ dilation=1,
+ causal_type='left'
+ )
+ )
+ )
+ self.convs1.apply(init_weights)
+ self.convs2.apply(init_weights)
+ self.activations1 = nn.ModuleList([
+ Snake(channels, alpha_logscale=False)
+ for _ in range(len(self.convs1))
+ ])
+ self.activations2 = nn.ModuleList([
+ Snake(channels, alpha_logscale=False)
+ for _ in range(len(self.convs2))
+ ])
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ for idx in range(len(self.convs1)):
+ xt = self.activations1[idx](x)
+ xt = self.convs1[idx](xt)
+ xt = self.activations2[idx](xt)
+ xt = self.convs2[idx](xt)
+ x = xt + x
+ return x
+
+ def remove_weight_norm(self):
+ for idx in range(len(self.convs1)):
+ remove_weight_norm(self.convs1[idx])
+ remove_weight_norm(self.convs2[idx])
+
+
+class SineGen(torch.nn.Module):
+ """ Definition of sine generator
+ SineGen(samp_rate, harmonic_num = 0,
+ sine_amp = 0.1, noise_std = 0.003,
+ voiced_threshold = 0,
+ flag_for_pulse=False)
+ samp_rate: sampling rate in Hz
+ harmonic_num: number of harmonic overtones (default 0)
+ sine_amp: amplitude of sine-wavefrom (default 0.1)
+ noise_std: std of Gaussian noise (default 0.003)
+ voiced_thoreshold: F0 threshold for U/V classification (default 0)
+ flag_for_pulse: this SinGen is used inside PulseGen (default False)
+ Note: when flag_for_pulse is True, the first time step of a voiced
+ segment is always sin(np.pi) or cos(0)
+ """
+
+ def __init__(self, samp_rate, harmonic_num=0,
+ sine_amp=0.1, noise_std=0.003,
+ voiced_threshold=0):
+ super(SineGen, self).__init__()
+ self.sine_amp = sine_amp
+ self.noise_std = noise_std
+ self.harmonic_num = harmonic_num
+ self.sampling_rate = samp_rate
+ self.voiced_threshold = voiced_threshold
+
+ def _f02uv(self, f0):
+ # generate uv signal
+ uv = (f0 > self.voiced_threshold).type(torch.float32)
+ return uv
+
+ @torch.no_grad()
+ def forward(self, f0):
+ """ sine_tensor, uv = forward(f0)
+ input F0: tensor(batchsize=1, dim=1, length)
+ f0 for unvoiced steps should be 0
+ output sine_tensor: tensor(batchsize=1, length, dim)
+ output uv: tensor(batchsize=1, length, 1)
+ """
+ f0 = f0.transpose(1, 2)
+ F_mat = torch.zeros((f0.size(0), self.harmonic_num + 1, f0.size(-1))).to(f0.device)
+ for i in range(self.harmonic_num + 1):
+ F_mat[:, i: i + 1, :] = f0 * (i + 1) / self.sampling_rate
+
+ theta_mat = 2 * np.pi * (torch.cumsum(F_mat, dim=-1) % 1)
+ u_dist = Uniform(low=-np.pi, high=np.pi)
+ phase_vec = u_dist.sample(sample_shape=(f0.size(0), self.harmonic_num + 1, 1)).to(F_mat.device)
+ phase_vec[:, 0, :] = 0
+
+ # generate sine waveforms
+ sine_waves = self.sine_amp * torch.sin(theta_mat + phase_vec)
+
+ # generate uv signal
+ uv = self._f02uv(f0)
+
+ # noise: for unvoiced should be similar to sine_amp
+ # std = self.sine_amp/3 -> max value ~ self.sine_amp
+ # . for voiced regions is self.noise_std
+ noise_amp = uv * self.noise_std + (1 - uv) * self.sine_amp / 3
+ noise = noise_amp * torch.randn_like(sine_waves)
+
+ # first: set the unvoiced part to 0 by uv
+ # then: additive noise
+ sine_waves = sine_waves * uv + noise
+ return sine_waves.transpose(1, 2), uv.transpose(1, 2), noise
+
+
+class SineGen2(torch.nn.Module):
+ """ Definition of sine generator
+ SineGen(samp_rate, harmonic_num = 0,
+ sine_amp = 0.1, noise_std = 0.003,
+ voiced_threshold = 0,
+ flag_for_pulse=False)
+ samp_rate: sampling rate in Hz
+ harmonic_num: number of harmonic overtones (default 0)
+ sine_amp: amplitude of sine-wavefrom (default 0.1)
+ noise_std: std of Gaussian noise (default 0.003)
+ voiced_thoreshold: F0 threshold for U/V classification (default 0)
+ flag_for_pulse: this SinGen is used inside PulseGen (default False)
+ Note: when flag_for_pulse is True, the first time step of a voiced
+ segment is always sin(np.pi) or cos(0)
+ """
+
+ def __init__(self, samp_rate, upsample_scale, harmonic_num=0,
+ sine_amp=0.1, noise_std=0.003,
+ voiced_threshold=0,
+ flag_for_pulse=False,
+ causal=False):
+ super(SineGen2, self).__init__()
+ self.sine_amp = sine_amp
+ self.noise_std = noise_std
+ self.harmonic_num = harmonic_num
+ self.dim = self.harmonic_num + 1
+ self.sampling_rate = samp_rate
+ self.voiced_threshold = voiced_threshold
+ self.flag_for_pulse = flag_for_pulse
+ self.upsample_scale = upsample_scale
+ self.causal = causal
+ if causal is True:
+ self.rand_ini = torch.rand(1, 9)
+ self.rand_ini[:, 0] = 0
+ self.sine_waves = torch.rand(1, 300 * 24000, 9)
+
+ def _f02uv(self, f0):
+ # generate uv signal
+ uv = (f0 > self.voiced_threshold).type(torch.float32)
+ return uv
+
+ def _f02sine(self, f0_values):
+ """ f0_values: (batchsize, length, dim)
+ where dim indicates fundamental tone and overtones
+ """
+ # convert to F0 in rad. The interger part n can be ignored
+ # because 2 * np.pi * n doesn't affect phase
+ rad_values = (f0_values / self.sampling_rate) % 1
+
+ # initial phase noise (no noise for fundamental component)
+ if self.training is False and self.causal is True:
+ rad_values[:, 0, :] = rad_values[:, 0, :] + self.rand_ini.to(rad_values.device)
+ else:
+ rand_ini = torch.rand(f0_values.shape[0], f0_values.shape[2], device=f0_values.device)
+ rand_ini[:, 0] = 0
+ rad_values[:, 0, :] = rad_values[:, 0, :] + rand_ini
+
+ # instantanouse phase sine[t] = sin(2*pi \sum_i=1 ^{t} rad)
+ if not self.flag_for_pulse:
+ rad_values = torch.nn.functional.interpolate(rad_values.transpose(1, 2),
+ scale_factor=1 / self.upsample_scale,
+ mode="linear").transpose(1, 2)
+
+ phase = torch.cumsum(rad_values, dim=1) * 2 * np.pi
+ phase = torch.nn.functional.interpolate(phase.transpose(1, 2) * self.upsample_scale,
+ scale_factor=self.upsample_scale, mode="nearest" if self.causal is True else 'linear').transpose(1, 2)
+ sines = torch.sin(phase)
+ else:
+ # If necessary, make sure that the first time step of every
+ # voiced segments is sin(pi) or cos(0)
+ # This is used for pulse-train generation
+
+ # identify the last time step in unvoiced segments
+ uv = self._f02uv(f0_values)
+ uv_1 = torch.roll(uv, shifts=-1, dims=1)
+ uv_1[:, -1, :] = 1
+ u_loc = (uv < 1) * (uv_1 > 0)
+
+ # get the instantanouse phase
+ tmp_cumsum = torch.cumsum(rad_values, dim=1)
+ # different batch needs to be processed differently
+ for idx in range(f0_values.shape[0]):
+ temp_sum = tmp_cumsum[idx, u_loc[idx, :, 0], :]
+ temp_sum[1:, :] = temp_sum[1:, :] - temp_sum[0:-1, :]
+ # stores the accumulation of i.phase within
+ # each voiced segments
+ tmp_cumsum[idx, :, :] = 0
+ tmp_cumsum[idx, u_loc[idx, :, 0], :] = temp_sum
+
+ # rad_values - tmp_cumsum: remove the accumulation of i.phase
+ # within the previous voiced segment.
+ i_phase = torch.cumsum(rad_values - tmp_cumsum, dim=1)
+
+ # get the sines
+ sines = torch.cos(i_phase * 2 * np.pi)
+ return sines
+
+ def forward(self, f0):
+ """ sine_tensor, uv = forward(f0)
+ input F0: tensor(batchsize=1, length, dim=1)
+ f0 for unvoiced steps should be 0
+ output sine_tensor: tensor(batchsize=1, length, dim)
+ output uv: tensor(batchsize=1, length, 1)
+ """
+ # fundamental component
+ fn = torch.multiply(f0, torch.FloatTensor([[range(1, self.harmonic_num + 2)]]).to(f0.device))
+
+ # generate sine waveforms
+ sine_waves = self._f02sine(fn) * self.sine_amp
+
+ # generate uv signal
+ uv = self._f02uv(f0)
+
+ # noise: for unvoiced should be similar to sine_amp
+ # std = self.sine_amp/3 -> max value ~ self.sine_amp
+ # . for voiced regions is self.noise_std
+ noise_amp = uv * self.noise_std + (1 - uv) * self.sine_amp / 3
+ if self.training is False and self.causal is True:
+ noise = noise_amp * self.sine_waves[:, :sine_waves.shape[1]].to(sine_waves.device)
+ else:
+ noise = noise_amp * torch.randn_like(sine_waves)
+
+ # first: set the unvoiced part to 0 by uv
+ # then: additive noise
+ sine_waves = sine_waves * uv + noise
+ return sine_waves, uv, noise
+
+
+class SourceModuleHnNSF(torch.nn.Module):
+ """ SourceModule for hn-nsf
+ SourceModule(sampling_rate, harmonic_num=0, sine_amp=0.1,
+ add_noise_std=0.003, voiced_threshod=0)
+ sampling_rate: sampling_rate in Hz
+ harmonic_num: number of harmonic above F0 (default: 0)
+ sine_amp: amplitude of sine source signal (default: 0.1)
+ add_noise_std: std of additive Gaussian noise (default: 0.003)
+ note that amplitude of noise in unvoiced is decided
+ by sine_amp
+ voiced_threshold: threhold to set U/V given F0 (default: 0)
+ Sine_source, noise_source = SourceModuleHnNSF(F0_sampled)
+ F0_sampled (batchsize, length, 1)
+ Sine_source (batchsize, length, 1)
+ noise_source (batchsize, length 1)
+ uv (batchsize, length, 1)
+ """
+
+ def __init__(self, sampling_rate, upsample_scale, harmonic_num=0, sine_amp=0.1,
+ add_noise_std=0.003, voiced_threshod=0, sinegen_type='1', causal=False):
+ super(SourceModuleHnNSF, self).__init__()
+
+ self.sine_amp = sine_amp
+ self.noise_std = add_noise_std
+
+ # to produce sine waveforms
+ if sinegen_type == '1':
+ self.l_sin_gen = SineGen(sampling_rate, harmonic_num, sine_amp, add_noise_std, voiced_threshod)
+ else:
+ self.l_sin_gen = SineGen2(sampling_rate, upsample_scale, harmonic_num, sine_amp, add_noise_std, voiced_threshod, causal=causal)
+
+ # to merge source harmonics into a single excitation
+ self.l_linear = torch.nn.Linear(harmonic_num + 1, 1)
+ self.l_tanh = torch.nn.Tanh()
+ self.causal = causal
+ if causal is True:
+ self.uv = torch.rand(1, 300 * 24000, 1)
+
+ def forward(self, x):
+ """
+ Sine_source, noise_source = SourceModuleHnNSF(F0_sampled)
+ F0_sampled (batchsize, length, 1)
+ Sine_source (batchsize, length, 1)
+ noise_source (batchsize, length 1)
+ """
+ # source for harmonic branch
+ with torch.no_grad():
+ sine_wavs, uv, _ = self.l_sin_gen(x)
+ sine_merge = self.l_tanh(self.l_linear(sine_wavs))
+
+ # source for noise branch, in the same shape as uv
+ if self.training is False and self.causal is True:
+ noise = self.uv[:, :uv.shape[1]] * self.sine_amp / 3
+ else:
+ noise = torch.randn_like(uv) * self.sine_amp / 3
+ return sine_merge, noise, uv
+
+
+class HiFTGenerator(nn.Module):
+ """
+ HiFTNet Generator: Neural Source Filter + ISTFTNet
+ https://arxiv.org/abs/2309.09493
+ """
+ def __init__(
+ self,
+ in_channels: int = 80,
+ base_channels: int = 512,
+ nb_harmonics: int = 8,
+ sampling_rate: int = 22050,
+ nsf_alpha: float = 0.1,
+ nsf_sigma: float = 0.003,
+ nsf_voiced_threshold: float = 10,
+ upsample_rates: List[int] = [8, 8],
+ upsample_kernel_sizes: List[int] = [16, 16],
+ istft_params: Dict[str, int] = {"n_fft": 16, "hop_len": 4},
+ resblock_kernel_sizes: List[int] = [3, 7, 11],
+ resblock_dilation_sizes: List[List[int]] = [[1, 3, 5], [1, 3, 5], [1, 3, 5]],
+ source_resblock_kernel_sizes: List[int] = [7, 11],
+ source_resblock_dilation_sizes: List[List[int]] = [[1, 3, 5], [1, 3, 5]],
+ lrelu_slope: float = 0.1,
+ audio_limit: float = 0.99,
+ f0_predictor: torch.nn.Module = None,
+ ):
+ super(HiFTGenerator, self).__init__()
+
+ self.out_channels = 1
+ self.nb_harmonics = nb_harmonics
+ self.sampling_rate = sampling_rate
+ self.istft_params = istft_params
+ self.lrelu_slope = lrelu_slope
+ self.audio_limit = audio_limit
+
+ self.num_kernels = len(resblock_kernel_sizes)
+ self.num_upsamples = len(upsample_rates)
+ # NOTE in CosyVoice2, we use the original SineGen implementation
+ self.m_source = SourceModuleHnNSF(
+ sampling_rate=sampling_rate,
+ upsample_scale=np.prod(upsample_rates) * istft_params["hop_len"],
+ harmonic_num=nb_harmonics,
+ sine_amp=nsf_alpha,
+ add_noise_std=nsf_sigma,
+ voiced_threshod=nsf_voiced_threshold,
+ sinegen_type='1' if self.sampling_rate == 22050 else '2',
+ causal=False)
+ self.f0_upsamp = torch.nn.Upsample(scale_factor=np.prod(upsample_rates) * istft_params["hop_len"])
+
+ self.conv_pre = weight_norm(
+ Conv1d(in_channels, base_channels, 7, 1, padding=3)
+ )
+
+ # Up
+ self.ups = nn.ModuleList()
+ for i, (u, k) in enumerate(zip(upsample_rates, upsample_kernel_sizes)):
+ self.ups.append(
+ weight_norm(
+ ConvTranspose1d(
+ base_channels // (2**i),
+ base_channels // (2**(i + 1)),
+ k,
+ u,
+ padding=(k - u) // 2,
+ )
+ )
+ )
+
+ # Down
+ self.source_downs = nn.ModuleList()
+ self.source_resblocks = nn.ModuleList()
+ downsample_rates = [1] + upsample_rates[::-1][:-1]
+ downsample_cum_rates = np.cumprod(downsample_rates)
+ for i, (u, k, d) in enumerate(zip(downsample_cum_rates[::-1], source_resblock_kernel_sizes, source_resblock_dilation_sizes)):
+ if u == 1:
+ self.source_downs.append(
+ Conv1d(istft_params["n_fft"] + 2, base_channels // (2 ** (i + 1)), 1, 1)
+ )
+ else:
+ self.source_downs.append(
+ Conv1d(istft_params["n_fft"] + 2, base_channels // (2 ** (i + 1)), u * 2, u, padding=(u // 2))
+ )
+
+ self.source_resblocks.append(
+ ResBlock(base_channels // (2 ** (i + 1)), k, d)
+ )
+
+ self.resblocks = nn.ModuleList()
+ for i in range(len(self.ups)):
+ ch = base_channels // (2**(i + 1))
+ for _, (k, d) in enumerate(zip(resblock_kernel_sizes, resblock_dilation_sizes)):
+ self.resblocks.append(ResBlock(ch, k, d))
+
+ self.conv_post = weight_norm(Conv1d(ch, istft_params["n_fft"] + 2, 7, 1, padding=3))
+ self.ups.apply(init_weights)
+ self.conv_post.apply(init_weights)
+ self.reflection_pad = nn.ReflectionPad1d((1, 0))
+ self.stft_window = torch.from_numpy(get_window("hann", istft_params["n_fft"], fftbins=True).astype(np.float32))
+ self.f0_predictor = f0_predictor
+
+ def remove_weight_norm(self):
+ print('Removing weight norm...')
+ for l in self.ups:
+ remove_weight_norm(l)
+ for l in self.resblocks:
+ l.remove_weight_norm()
+ remove_weight_norm(self.conv_pre)
+ remove_weight_norm(self.conv_post)
+ self.m_source.remove_weight_norm()
+ for l in self.source_downs:
+ remove_weight_norm(l)
+ for l in self.source_resblocks:
+ l.remove_weight_norm()
+
+ def _stft(self, x):
+ spec = torch.stft(
+ x,
+ self.istft_params["n_fft"], self.istft_params["hop_len"], self.istft_params["n_fft"], window=self.stft_window.to(x.device),
+ return_complex=True)
+ spec = torch.view_as_real(spec) # [B, F, TT, 2]
+ return spec[..., 0], spec[..., 1]
+
+ def _istft(self, magnitude, phase):
+ magnitude = torch.clip(magnitude, max=1e2)
+ real = magnitude * torch.cos(phase)
+ img = magnitude * torch.sin(phase)
+ inverse_transform = torch.istft(torch.complex(real, img), self.istft_params["n_fft"], self.istft_params["hop_len"],
+ self.istft_params["n_fft"], window=self.stft_window.to(magnitude.device))
+ return inverse_transform
+
+ def decode(self, x: torch.Tensor, s: torch.Tensor = torch.zeros(1, 1, 0)) -> torch.Tensor:
+ s_stft_real, s_stft_imag = self._stft(s.squeeze(1))
+ s_stft = torch.cat([s_stft_real, s_stft_imag], dim=1)
+
+ x = self.conv_pre(x)
+ for i in range(self.num_upsamples):
+ x = F.leaky_relu(x, self.lrelu_slope)
+ x = self.ups[i](x)
+
+ if i == self.num_upsamples - 1:
+ x = self.reflection_pad(x)
+
+ # fusion
+ si = self.source_downs[i](s_stft)
+ si = self.source_resblocks[i](si)
+ x = x + si
+
+ xs = None
+ for j in range(self.num_kernels):
+ if xs is None:
+ xs = self.resblocks[i * self.num_kernels + j](x)
+ else:
+ xs += self.resblocks[i * self.num_kernels + j](x)
+ x = xs / self.num_kernels
+
+ x = F.leaky_relu(x)
+ x = self.conv_post(x)
+ magnitude = torch.exp(x[:, :self.istft_params["n_fft"] // 2 + 1, :])
+ phase = torch.sin(x[:, self.istft_params["n_fft"] // 2 + 1:, :]) # actually, sin is redundancy
+
+ x = self._istft(magnitude, phase)
+ x = torch.clamp(x, -self.audio_limit, self.audio_limit)
+ return x
+
+ def forward(
+ self,
+ batch: dict,
+ device: torch.device,
+ ) -> Dict[str, Optional[torch.Tensor]]:
+ speech_feat = batch['speech_feat'].transpose(1, 2).to(device)
+ # mel->f0
+ f0 = self.f0_predictor(speech_feat)
+ # f0->source
+ s = self.f0_upsamp(f0[:, None]).transpose(1, 2) # bs,n,t
+ s, _, _ = self.m_source(s)
+ s = s.transpose(1, 2)
+ # mel+source->speech
+ generated_speech = self.decode(x=speech_feat, s=s)
+ return generated_speech, f0
+
+ @torch.inference_mode()
+ def inference(self, speech_feat: torch.Tensor, cache_source: torch.Tensor = torch.zeros(1, 1, 0)) -> torch.Tensor:
+ # mel->f0
+ f0 = self.f0_predictor(speech_feat)
+ # f0->source
+ s = self.f0_upsamp(f0[:, None]).transpose(1, 2) # bs,n,t
+ s, _, _ = self.m_source(s)
+ s = s.transpose(1, 2)
+ # use cache_source to avoid glitch
+ if cache_source.shape[2] != 0:
+ s[:, :, :cache_source.shape[2]] = cache_source
+ generated_speech = self.decode(x=speech_feat, s=s)
+ return generated_speech, s
+
+
+class CausalHiFTGenerator(HiFTGenerator):
+ """
+ HiFTNet Generator: Neural Source Filter + ISTFTNet
+ https://arxiv.org/abs/2309.09493
+ """
+ def __init__(
+ self,
+ in_channels: int = 80,
+ base_channels: int = 512,
+ nb_harmonics: int = 8,
+ sampling_rate: int = 22050,
+ nsf_alpha: float = 0.1,
+ nsf_sigma: float = 0.003,
+ nsf_voiced_threshold: float = 10,
+ upsample_rates: List[int] = [8, 8],
+ upsample_kernel_sizes: List[int] = [16, 16],
+ istft_params: Dict[str, int] = {"n_fft": 16, "hop_len": 4},
+ resblock_kernel_sizes: List[int] = [3, 7, 11],
+ resblock_dilation_sizes: List[List[int]] = [[1, 3, 5], [1, 3, 5], [1, 3, 5]],
+ source_resblock_kernel_sizes: List[int] = [7, 11],
+ source_resblock_dilation_sizes: List[List[int]] = [[1, 3, 5], [1, 3, 5]],
+ lrelu_slope: float = 0.1,
+ audio_limit: float = 0.99,
+ conv_pre_look_right: int = 4,
+ f0_predictor: torch.nn.Module = None,
+ ):
+ torch.nn.Module.__init__(self)
+
+ self.out_channels = 1
+ self.nb_harmonics = nb_harmonics
+ self.sampling_rate = sampling_rate
+ self.istft_params = istft_params
+ self.lrelu_slope = lrelu_slope
+ self.audio_limit = audio_limit
+
+ self.num_kernels = len(resblock_kernel_sizes)
+ self.num_upsamples = len(upsample_rates)
+ self.m_source = SourceModuleHnNSF(
+ sampling_rate=sampling_rate,
+ upsample_scale=np.prod(upsample_rates) * istft_params["hop_len"],
+ harmonic_num=nb_harmonics,
+ sine_amp=nsf_alpha,
+ add_noise_std=nsf_sigma,
+ voiced_threshod=nsf_voiced_threshold,
+ sinegen_type='1' if self.sampling_rate == 22050 else '2',
+ causal=True)
+ self.upsample_rates = upsample_rates
+ self.f0_upsamp = torch.nn.Upsample(scale_factor=np.prod(upsample_rates) * istft_params["hop_len"])
+
+ self.conv_pre = weight_norm(
+ CausalConv1d(in_channels, base_channels, conv_pre_look_right + 1, 1, causal_type='right')
+ )
+
+ # Up
+ self.ups = nn.ModuleList()
+ for i, (u, k) in enumerate(zip(upsample_rates, upsample_kernel_sizes)):
+ self.ups.append(
+ weight_norm(
+ CausalConv1dUpsample(
+ base_channels // (2**i),
+ base_channels // (2**(i + 1)),
+ k,
+ u,
+ )
+ )
+ )
+
+ # Down
+ self.source_downs = nn.ModuleList()
+ self.source_resblocks = nn.ModuleList()
+ downsample_rates = [1] + upsample_rates[::-1][:-1]
+ downsample_cum_rates = np.cumprod(downsample_rates)
+ for i, (u, k, d) in enumerate(zip(downsample_cum_rates[::-1], source_resblock_kernel_sizes, source_resblock_dilation_sizes)):
+ if u == 1:
+ self.source_downs.append(
+ CausalConv1d(istft_params["n_fft"] + 2, base_channels // (2 ** (i + 1)), 1, 1, causal_type='left')
+ )
+ else:
+ self.source_downs.append(
+ CausalConv1dDownSample(istft_params["n_fft"] + 2, base_channels // (2 ** (i + 1)), u * 2, u)
+ )
+
+ self.source_resblocks.append(
+ ResBlock(base_channels // (2 ** (i + 1)), k, d, causal=True)
+ )
+
+ self.resblocks = nn.ModuleList()
+ for i in range(len(self.ups)):
+ ch = base_channels // (2**(i + 1))
+ for _, (k, d) in enumerate(zip(resblock_kernel_sizes, resblock_dilation_sizes)):
+ self.resblocks.append(ResBlock(ch, k, d, causal=True))
+
+ self.conv_post = weight_norm(CausalConv1d(ch, istft_params["n_fft"] + 2, 7, 1, causal_type='left'))
+ self.ups.apply(init_weights)
+ self.conv_post.apply(init_weights)
+ self.reflection_pad = nn.ReflectionPad1d((1, 0))
+ self.stft_window = torch.from_numpy(get_window("hann", istft_params["n_fft"], fftbins=True).astype(np.float32))
+ self.conv_pre_look_right = conv_pre_look_right
+ self.f0_predictor = f0_predictor
+
+ def decode(self, x: torch.Tensor, s: torch.Tensor = torch.zeros(1, 1, 0), finalize: bool = True) -> torch.Tensor:
+ s_stft_real, s_stft_imag = self._stft(s.squeeze(1))
+ if finalize is True:
+ x = self.conv_pre(x)
+ else:
+ x = self.conv_pre(x[:, :, :-self.conv_pre_look_right], x[:, :, -self.conv_pre_look_right:])
+ s_stft_real = s_stft_real[:, :, :-int(np.prod(self.upsample_rates) * self.conv_pre_look_right)]
+ s_stft_imag = s_stft_imag[:, :, :-int(np.prod(self.upsample_rates) * self.conv_pre_look_right)]
+ s_stft = torch.cat([s_stft_real, s_stft_imag], dim=1)
+
+ for i in range(self.num_upsamples):
+ x = F.leaky_relu(x, self.lrelu_slope)
+ x = self.ups[i](x)
+
+ if i == self.num_upsamples - 1:
+ x = self.reflection_pad(x)
+
+ # fusion
+ si = self.source_downs[i](s_stft)
+ si = self.source_resblocks[i](si)
+ x = x + si
+
+ xs = None
+ for j in range(self.num_kernels):
+ if xs is None:
+ xs = self.resblocks[i * self.num_kernels + j](x)
+ else:
+ xs += self.resblocks[i * self.num_kernels + j](x)
+ x = xs / self.num_kernels
+
+ x = F.leaky_relu(x)
+ x = self.conv_post(x)
+ magnitude = torch.exp(x[:, :self.istft_params["n_fft"] // 2 + 1, :])
+ phase = torch.sin(x[:, self.istft_params["n_fft"] // 2 + 1:, :]) # actually, sin is redundancy
+
+ x = self._istft(magnitude, phase)
+ if finalize is False:
+ x = x[:, :-int(np.prod(self.upsample_rates) * self.istft_params['hop_len'])]
+ x = torch.clamp(x, -self.audio_limit, self.audio_limit)
+ return x
+
+ @torch.inference_mode()
+ def inference(self, speech_feat: torch.Tensor, finalize: bool = True) -> torch.Tensor:
+ # mel->f0 NOTE f0_predictor precision is crucial for causal inference, move self.f0_predictor to cpu if necessary
+ self.f0_predictor.to(torch.float64)
+ f0 = self.f0_predictor(speech_feat.to(torch.float64), finalize=finalize).to(speech_feat)
+ # f0->source
+ s = self.f0_upsamp(f0[:, None]).transpose(1, 2) # bs,n,t
+ s, _, _ = self.m_source(s)
+ s = s.transpose(1, 2)
+ if finalize is True:
+ generated_speech = self.decode(x=speech_feat, s=s, finalize=finalize)
+ else:
+ generated_speech = self.decode(x=speech_feat[:, :, :-self.f0_predictor.condnet[0].causal_padding], s=s, finalize=finalize)
+ return generated_speech, s
diff --git a/freevideo_engine/dubbing/cosyvoice3/llm/__init__.py b/freevideo_engine/dubbing/cosyvoice3/llm/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/freevideo_engine/dubbing/cosyvoice3/llm/llm.py b/freevideo_engine/dubbing/cosyvoice3/llm/llm.py
new file mode 100644
index 0000000..6874fbb
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/llm/llm.py
@@ -0,0 +1,119 @@
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Zhihao Du)
+# 2025 Alibaba Inc (authors: Xiang Lyu, Yabin Li, Qihua, Shengqiang Li)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Modified by FreeVideo: reduced to the CosyVoice3 inference path (no training, streaming
+# or vLLM code); the Qwen2 backbone is built from its config and filled from llm.pt.
+from typing import Callable, Generator, List
+
+import torch
+from torch import nn
+from transformers import AutoConfig, Qwen2ForCausalLM
+
+
+class Qwen2Encoder(torch.nn.Module):
+ def __init__(self, config_dir):
+ super().__init__()
+ config = AutoConfig.from_pretrained(config_dir)
+ config.dtype = torch.float32
+ self.model = Qwen2ForCausalLM(config).to(torch.float32)
+
+ def forward_one_step(self, xs, cache=None):
+ # single sequence, no padding: the causal mask comes from the cache.
+ # (a one-step mask of length 1 would be applied against the whole KV cache by transformers>=5)
+ outs = self.model(
+ inputs_embeds=xs,
+ attention_mask=None,
+ output_hidden_states=True,
+ return_dict=True,
+ use_cache=True,
+ past_key_values=cache,
+ )
+ return outs.hidden_states[-1], outs.past_key_values
+
+
+class CosyVoice3LM(torch.nn.Module):
+ def __init__(
+ self,
+ llm_input_size: int,
+ llm_output_size: int,
+ speech_token_size: int,
+ llm: torch.nn.Module,
+ sampling: Callable,
+ mix_ratio: List[int] = [5, 15],
+ ):
+ super().__init__()
+ self.llm_input_size = llm_input_size
+ self.llm_output_size = llm_output_size
+ self.speech_token_size = speech_token_size
+ self.sos = speech_token_size + 0
+ self.eos_token = speech_token_size + 1
+ self.task_id = speech_token_size + 2
+ self.fill_token = speech_token_size + 3
+
+ self.llm = llm
+ self.llm_decoder = nn.Linear(llm_output_size, speech_token_size + 200, bias=False)
+ self.speech_embedding = torch.nn.Embedding(speech_token_size + 200, llm_input_size)
+ self.sampling = sampling
+ self.mix_ratio = mix_ratio
+ self.stop_token_ids = [speech_token_size + i for i in range(200)]
+
+ def sampling_ids(self, weighted_scores: torch.Tensor, decoded_tokens: List, sampling: int, ignore_eos: bool = True):
+ if ignore_eos is True:
+ weighted_scores[self.speech_token_size] = -float('inf')
+ return self.sampling(weighted_scores, decoded_tokens, sampling)
+
+ @torch.inference_mode()
+ def inference(
+ self,
+ text: torch.Tensor,
+ text_len: torch.Tensor,
+ prompt_text: torch.Tensor,
+ prompt_text_len: torch.Tensor,
+ prompt_speech_token: torch.Tensor,
+ prompt_speech_token_len: torch.Tensor,
+ embedding: torch.Tensor,
+ sampling: int = 25,
+ max_token_text_ratio: float = 20,
+ min_token_text_ratio: float = 2,
+ uuid: str = '',
+ ) -> Generator[int, None, None]:
+ device = text.device
+ text = torch.concat([prompt_text, text], dim=1)
+ text_len += prompt_text_len
+ text_emb = self.llm.model.model.embed_tokens(text)
+ # 151646 is the <|endofprompt|> token
+ assert 151646 in text, '<|endofprompt|> not detected in CosyVoice3 text or prompt_text, check your input!'
+
+ sos_emb = self.speech_embedding.weight[self.sos].reshape(1, 1, -1)
+ task_id_emb = self.speech_embedding.weight[self.task_id].reshape(1, 1, -1)
+ if prompt_speech_token_len != 0:
+ prompt_speech_token_emb = self.speech_embedding(prompt_speech_token)
+ else:
+ prompt_speech_token_emb = torch.zeros(1, 0, self.llm_input_size, dtype=text_emb.dtype).to(device)
+ lm_input = torch.concat([sos_emb, text_emb, task_id_emb, prompt_speech_token_emb], dim=1)
+
+ min_len = int((text_len - prompt_text_len) * min_token_text_ratio)
+ max_len = int((text_len - prompt_text_len) * max_token_text_ratio)
+
+ out_tokens = []
+ cache = None
+ for i in range(max_len):
+ y_pred, cache = self.llm.forward_one_step(lm_input, cache=cache)
+ logp = self.llm_decoder(y_pred[:, -1]).log_softmax(dim=-1)
+ top_ids = self.sampling_ids(logp.squeeze(dim=0), out_tokens, sampling, ignore_eos=True if i < min_len else False)
+ if top_ids in self.stop_token_ids:
+ break
+ yield top_ids
+ out_tokens.append(top_ids)
+ lm_input = self.speech_embedding.weight[top_ids].reshape(1, 1, -1)
diff --git a/freevideo_engine/dubbing/cosyvoice3/tokenizer/__init__.py b/freevideo_engine/dubbing/cosyvoice3/tokenizer/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/freevideo_engine/dubbing/cosyvoice3/tokenizer/tokenizer.py b/freevideo_engine/dubbing/cosyvoice3/tokenizer/tokenizer.py
new file mode 100644
index 0000000..3fc95a5
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/tokenizer/tokenizer.py
@@ -0,0 +1,107 @@
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Modified by FreeVideo: reduced to the Qwen-based CosyVoice tokenizers (no tiktoken).
+from functools import lru_cache
+
+import torch
+from transformers import AutoTokenizer
+
+
+class CosyVoice2Tokenizer():
+ def __init__(self, token_path, skip_special_tokens=True):
+ super().__init__()
+ # NOTE: non-chat model, all these special tokens keep randomly initialized.
+ special_tokens = {
+ 'eos_token': '<|endoftext|>',
+ 'pad_token': '<|endoftext|>',
+ 'additional_special_tokens': [
+ '<|im_start|>', '<|im_end|>', '<|endofprompt|>',
+ '[breath]', '<strong>', '</strong>', '[noise]',
+ '[laughter]', '[cough]', '[clucking]', '[accent]',
+ '[quick_breath]',
+ "<laughter>", "</laughter>",
+ "[hissing]", "[sigh]", "[vocalized-noise]",
+ "[lipsmack]", "[mn]"
+ ]
+ }
+ self.special_tokens = special_tokens
+ self.tokenizer = AutoTokenizer.from_pretrained(token_path)
+ self.tokenizer.add_special_tokens(special_tokens)
+ self.skip_special_tokens = skip_special_tokens
+
+ def encode(self, text, **kwargs):
+ tokens = self.tokenizer([text], return_tensors="pt")
+ tokens = tokens["input_ids"][0].cpu().tolist()
+ return tokens
+
+ def decode(self, tokens):
+ tokens = torch.tensor(tokens, dtype=torch.int64)
+ text = self.tokenizer.batch_decode([tokens], skip_special_tokens=self.skip_special_tokens)[0]
+ return text
+
+
+class CosyVoice3Tokenizer(CosyVoice2Tokenizer):
+ def __init__(self, token_path, skip_special_tokens=True):
+ # NOTE: non-chat model, all these special tokens keep randomly initialized.
+ special_tokens = {
+ 'eos_token': '<|endoftext|>',
+ 'pad_token': '<|endoftext|>',
+ 'additional_special_tokens': [
+ '<|im_start|>', '<|im_end|>', '<|endofprompt|>',
+ '[breath]', '<strong>', '</strong>', '[noise]',
+ '[laughter]', '[cough]', '[clucking]', '[accent]',
+ '[quick_breath]',
+ "<laughter>", "</laughter>",
+ "[hissing]", "[sigh]", "[vocalized-noise]",
+ "[lipsmack]", "[mn]", "<|endofsystem|>",
+ "[AA]", "[AA0]", "[AA1]", "[AA2]", "[AE]", "[AE0]", "[AE1]", "[AE2]", "[AH]", "[AH0]", "[AH1]", "[AH2]",
+ "[AO]", "[AO0]", "[AO1]", "[AO2]", "[AW]", "[AW0]", "[AW1]", "[AW2]", "[AY]", "[AY0]", "[AY1]", "[AY2]",
+ "[B]", "[CH]", "[D]", "[DH]", "[EH]", "[EH0]", "[EH1]", "[EH2]", "[ER]", "[ER0]", "[ER1]", "[ER2]", "[EY]",
+ "[EY0]", "[EY1]", "[EY2]", "[F]", "[G]", "[HH]", "[IH]", "[IH0]", "[IH1]", "[IH2]", "[IY]", "[IY0]", "[IY1]",
+ "[IY2]", "[JH]", "[K]", "[L]", "[M]", "[N]", "[NG]", "[OW]", "[OW0]", "[OW1]", "[OW2]", "[OY]", "[OY0]",
+ "[OY1]", "[OY2]", "[P]", "[R]", "[S]", "[SH]", "[T]", "[TH]", "[UH]", "[UH0]", "[UH1]", "[UH2]", "[UW]",
+ "[UW0]", "[UW1]", "[UW2]", "[V]", "[W]", "[Y]", "[Z]", "[ZH]",
+ "[a]", "[ai]", "[an]", "[ang]", "[ao]", "[b]", "[c]", "[ch]", "[d]", "[e]", "[ei]", "[en]", "[eng]", "[f]",
+ "[g]", "[h]", "[i]", "[ian]", "[in]", "[ing]", "[iu]", "[ià]", "[iàn]", "[iàng]", "[iào]", "[iá]", "[ián]",
+ "[iáng]", "[iáo]", "[iè]", "[ié]", "[iòng]", "[ióng]", "[iù]", "[iú]", "[iā]", "[iān]", "[iāng]", "[iāo]",
+ "[iē]", "[iě]", "[iōng]", "[iū]", "[iǎ]", "[iǎn]", "[iǎng]", "[iǎo]", "[iǒng]", "[iǔ]", "[j]", "[k]", "[l]",
+ "[m]", "[n]", "[o]", "[ong]", "[ou]", "[p]", "[q]", "[r]", "[s]", "[sh]", "[t]", "[u]", "[uang]", "[ue]",
+ "[un]", "[uo]", "[uà]", "[uài]", "[uàn]", "[uàng]", "[uá]", "[uái]", "[uán]", "[uáng]", "[uè]", "[ué]", "[uì]",
+ "[uí]", "[uò]", "[uó]", "[uā]", "[uāi]", "[uān]", "[uāng]", "[uē]", "[uě]", "[uī]", "[uō]", "[uǎ]", "[uǎi]",
+ "[uǎn]", "[uǎng]", "[uǐ]", "[uǒ]", "[vè]", "[w]", "[x]", "[y]", "[z]", "[zh]", "[à]", "[ài]", "[àn]", "[àng]",
+ "[ào]", "[á]", "[ái]", "[án]", "[áng]", "[áo]", "[è]", "[èi]", "[èn]", "[èng]", "[èr]", "[é]", "[éi]", "[én]",
+ "[éng]", "[ér]", "[ì]", "[ìn]", "[ìng]", "[í]", "[ín]", "[íng]", "[ò]", "[òng]", "[òu]", "[ó]", "[óng]", "[óu]",
+ "[ù]", "[ùn]", "[ú]", "[ún]", "[ā]", "[āi]", "[ān]", "[āng]", "[āo]", "[ē]", "[ēi]", "[ēn]", "[ēng]", "[ě]",
+ "[ěi]", "[ěn]", "[ěng]", "[ěr]", "[ī]", "[īn]", "[īng]", "[ō]", "[ōng]", "[ōu]", "[ū]", "[ūn]", "[ǎ]", "[ǎi]",
+ "[ǎn]", "[ǎng]", "[ǎo]", "[ǐ]", "[ǐn]", "[ǐng]", "[ǒ]", "[ǒng]", "[ǒu]", "[ǔ]", "[ǔn]", "[ǘ]", "[ǚ]", "[ǜ]"
+ ]
+ }
+ self.special_tokens = special_tokens
+ self.tokenizer = AutoTokenizer.from_pretrained(token_path)
+ self.tokenizer.add_special_tokens(special_tokens)
+ self.skip_special_tokens = skip_special_tokens
+
+
+@lru_cache(maxsize=None)
+def get_qwen_tokenizer(
+ token_path: str,
+ skip_special_tokens: bool,
+ version: str = 'cosyvoice2'
+):
+ if version == 'cosyvoice2':
+ return CosyVoice2Tokenizer(token_path=token_path, skip_special_tokens=skip_special_tokens)
+ elif version == 'cosyvoice3':
+ return CosyVoice3Tokenizer(token_path=token_path, skip_special_tokens=skip_special_tokens)
+ else:
+ raise ValueError
diff --git a/freevideo_engine/dubbing/cosyvoice3/transformer/__init__.py b/freevideo_engine/dubbing/cosyvoice3/transformer/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/freevideo_engine/dubbing/cosyvoice3/transformer/activation.py b/freevideo_engine/dubbing/cosyvoice3/transformer/activation.py
new file mode 100644
index 0000000..8cea548
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/transformer/activation.py
@@ -0,0 +1,84 @@
+# Copyright (c) 2020 Johns Hopkins University (Shinji Watanabe)
+# 2020 Northwestern Polytechnical University (Pengcheng Guo)
+# 2020 Mobvoi Inc (Binbin Zhang)
+# 2024 Alibaba Inc (Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Swish() activation function for Conformer."""
+
+import torch
+from torch import nn, sin, pow
+from torch.nn import Parameter
+
+
+class Swish(torch.nn.Module):
+ """Construct an Swish object."""
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ """Return Swish activation function."""
+ return x * torch.sigmoid(x)
+
+
+# Implementation adapted from https://github.com/EdwardDixon/snake under the MIT license.
+# LICENSE is in incl_licenses directory.
+class Snake(nn.Module):
+ '''
+ Implementation of a sine-based periodic activation function
+ Shape:
+ - Input: (B, C, T)
+ - Output: (B, C, T), same shape as the input
+ Parameters:
+ - alpha - trainable parameter
+ References:
+ - This activation function is from this paper by Liu Ziyin, Tilman Hartwig, Masahito Ueda:
+ https://arxiv.org/abs/2006.08195
+ Examples:
+ >>> a1 = snake(256)
+ >>> x = torch.randn(256)
+ >>> x = a1(x)
+ '''
+ def __init__(self, in_features, alpha=1.0, alpha_trainable=True, alpha_logscale=False):
+ '''
+ Initialization.
+ INPUT:
+ - in_features: shape of the input
+ - alpha: trainable parameter
+ alpha is initialized to 1 by default, higher values = higher-frequency.
+ alpha will be trained along with the rest of your model.
+ '''
+ super(Snake, self).__init__()
+ self.in_features = in_features
+
+ # initialize alpha
+ self.alpha_logscale = alpha_logscale
+ if self.alpha_logscale: # log scale alphas initialized to zeros
+ self.alpha = Parameter(torch.zeros(in_features) * alpha)
+ else: # linear scale alphas initialized to ones
+ self.alpha = Parameter(torch.ones(in_features) * alpha)
+
+ self.alpha.requires_grad = alpha_trainable
+
+ self.no_div_by_zero = 0.000000001
+
+ def forward(self, x):
+ '''
+ Forward pass of the function.
+ Applies the function to the input elementwise.
+ Snake ∶= x + 1/a * sin^2 (xa)
+ '''
+ alpha = self.alpha.unsqueeze(0).unsqueeze(-1) # line up with x to [B, C, T]
+ if self.alpha_logscale:
+ alpha = torch.exp(alpha)
+ x = x + (1.0 / (alpha + self.no_div_by_zero)) * pow(sin(x * alpha), 2)
+
+ return x
diff --git a/freevideo_engine/dubbing/cosyvoice3/transformer/convolution.py b/freevideo_engine/dubbing/cosyvoice3/transformer/convolution.py
new file mode 100644
index 0000000..edb32b2
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/transformer/convolution.py
@@ -0,0 +1,258 @@
+# Copyright (c) 2020 Mobvoi Inc. (authors: Binbin Zhang, Di Wu)
+# 2024 Alibaba Inc (Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Modified from ESPnet(https://github.com/espnet/espnet)
+"""ConvolutionModule definition."""
+
+from typing import Tuple
+
+import torch
+from torch import nn
+import torch.nn.functional as F
+
+
+class ConvolutionModule(nn.Module):
+ """ConvolutionModule in Conformer model."""
+
+ def __init__(self,
+ channels: int,
+ kernel_size: int = 15,
+ activation: nn.Module = nn.ReLU(),
+ norm: str = "batch_norm",
+ causal: bool = False,
+ bias: bool = True):
+ """Construct an ConvolutionModule object.
+ Args:
+ channels (int): The number of channels of conv layers.
+ kernel_size (int): Kernel size of conv layers.
+ causal (int): Whether use causal convolution or not
+ """
+ super().__init__()
+
+ self.pointwise_conv1 = nn.Conv1d(
+ channels,
+ 2 * channels,
+ kernel_size=1,
+ stride=1,
+ padding=0,
+ bias=bias,
+ )
+ # self.lorder is used to distinguish if it's a causal convolution,
+ # if self.lorder > 0: it's a causal convolution, the input will be
+ # padded with self.lorder frames on the left in forward.
+ # else: it's a symmetrical convolution
+ if causal:
+ padding = 0
+ self.lorder = kernel_size - 1
+ else:
+ # kernel_size should be an odd number for none causal convolution
+ assert (kernel_size - 1) % 2 == 0
+ padding = (kernel_size - 1) // 2
+ self.lorder = 0
+ self.depthwise_conv = nn.Conv1d(
+ channels,
+ channels,
+ kernel_size,
+ stride=1,
+ padding=padding,
+ groups=channels,
+ bias=bias,
+ )
+
+ assert norm in ['batch_norm', 'layer_norm']
+ if norm == "batch_norm":
+ self.use_layer_norm = False
+ self.norm = nn.BatchNorm1d(channels)
+ else:
+ self.use_layer_norm = True
+ self.norm = nn.LayerNorm(channels)
+
+ self.pointwise_conv2 = nn.Conv1d(
+ channels,
+ channels,
+ kernel_size=1,
+ stride=1,
+ padding=0,
+ bias=bias,
+ )
+ self.activation = activation
+
+ def forward(
+ self,
+ x: torch.Tensor,
+ mask_pad: torch.Tensor = torch.ones((0, 0, 0), dtype=torch.bool),
+ cache: torch.Tensor = torch.zeros((0, 0, 0)),
+ ) -> Tuple[torch.Tensor, torch.Tensor]:
+ """Compute convolution module.
+ Args:
+ x (torch.Tensor): Input tensor (#batch, time, channels).
+ mask_pad (torch.Tensor): used for batch padding (#batch, 1, time),
+ (0, 0, 0) means fake mask.
+ cache (torch.Tensor): left context cache, it is only
+ used in causal convolution (#batch, channels, cache_t),
+ (0, 0, 0) meas fake cache.
+ Returns:
+ torch.Tensor: Output tensor (#batch, time, channels).
+ """
+ # exchange the temporal dimension and the feature dimension
+ x = x.transpose(1, 2) # (#batch, channels, time)
+
+ # mask batch padding
+ if mask_pad.size(2) > 0: # time > 0
+ x.masked_fill_(~mask_pad, 0.0)
+
+ if self.lorder > 0:
+ if cache.size(2) == 0: # cache_t == 0
+ x = nn.functional.pad(x, (self.lorder, 0), 'constant', 0.0)
+ else:
+ assert cache.size(0) == x.size(0) # equal batch
+ assert cache.size(1) == x.size(1) # equal channel
+ x = torch.cat((cache, x), dim=2)
+ assert (x.size(2) > self.lorder)
+ new_cache = x[:, :, -self.lorder:]
+ else:
+ # It's better we just return None if no cache is required,
+ # However, for JIT export, here we just fake one tensor instead of
+ # None.
+ new_cache = torch.zeros((0, 0, 0), dtype=x.dtype, device=x.device)
+
+ # GLU mechanism
+ x = self.pointwise_conv1(x) # (batch, 2*channel, dim)
+ x = nn.functional.glu(x, dim=1) # (batch, channel, dim)
+
+ # 1D Depthwise Conv
+ x = self.depthwise_conv(x)
+ if self.use_layer_norm:
+ x = x.transpose(1, 2)
+ x = self.activation(self.norm(x))
+ if self.use_layer_norm:
+ x = x.transpose(1, 2)
+ x = self.pointwise_conv2(x)
+ # mask batch padding
+ if mask_pad.size(2) > 0: # time > 0
+ x.masked_fill_(~mask_pad, 0.0)
+
+ return x.transpose(1, 2), new_cache
+
+
+# NOTE(Xiang Lyu) causal conv module used in convolution-based vocoder
+class CausalConv1d(torch.nn.Conv1d):
+ def __init__(
+ self,
+ in_channels: int,
+ out_channels: int,
+ kernel_size: int,
+ stride: int = 1,
+ dilation: int = 1,
+ groups: int = 1,
+ bias: bool = True,
+ padding_mode: str = 'zeros',
+ causal_type: str = 'left',
+ device=None,
+ dtype=None
+ ) -> None:
+ super(CausalConv1d, self).__init__(in_channels, out_channels,
+ kernel_size, stride=1,
+ padding=0, dilation=dilation,
+ groups=groups, bias=bias,
+ padding_mode=padding_mode,
+ device=device, dtype=dtype)
+ assert stride == 1
+ self.causal_padding = int((kernel_size * dilation - dilation) / 2) * 2 + (kernel_size + 1) % 2
+ assert causal_type in ['left', 'right']
+ self.causal_type = causal_type
+
+ def forward(self, x: torch.Tensor, cache: torch.Tensor = torch.zeros(0, 0, 0)) -> Tuple[torch.Tensor]:
+ input_timestep = x.shape[2]
+ if cache.size(2) == 0:
+ cache = torch.zeros(x.shape[0], x.shape[1], self.causal_padding).to(x)
+ assert cache.size(2) == self.causal_padding
+ if self.causal_type == 'left':
+ x = torch.concat([cache, x], dim=2)
+ else:
+ x = torch.concat([x, cache], dim=2)
+ x = super(CausalConv1d, self).forward(x)
+ assert x.shape[2] == input_timestep
+ return x
+
+
+class CausalConv1dDownSample(torch.nn.Conv1d):
+ def __init__(
+ self,
+ in_channels: int,
+ out_channels: int,
+ kernel_size: int,
+ stride: int = 1,
+ dilation: int = 1,
+ groups: int = 1,
+ bias: bool = True,
+ padding_mode: str = 'zeros',
+ device=None,
+ dtype=None
+ ) -> None:
+ super(CausalConv1dDownSample, self).__init__(in_channels, out_channels,
+ kernel_size, stride,
+ padding=0, dilation=dilation,
+ groups=groups, bias=bias,
+ padding_mode=padding_mode,
+ device=device, dtype=dtype)
+ assert stride != 1 and dilation == 1
+ assert kernel_size % stride == 0
+ self.causal_padding = stride - 1
+
+ def forward(self, x: torch.Tensor, cache: torch.Tensor = torch.zeros(0, 0, 0)) -> Tuple[torch.Tensor, torch.Tensor]:
+ if cache.size(2) == 0:
+ x = F.pad(x, (self.causal_padding, 0), value=0.0)
+ else:
+ assert cache.size(2) == self.causal_padding
+ x = torch.concat([cache, x], dim=2)
+ x = super(CausalConv1dDownSample, self).forward(x)
+ return x
+
+
+class CausalConv1dUpsample(torch.nn.Conv1d):
+ def __init__(
+ self,
+ in_channels: int,
+ out_channels: int,
+ kernel_size: int,
+ stride: int = 1,
+ dilation: int = 1,
+ groups: int = 1,
+ bias: bool = True,
+ padding_mode: str = 'zeros',
+ device=None,
+ dtype=None
+ ) -> None:
+ super(CausalConv1dUpsample, self).__init__(in_channels, out_channels,
+ kernel_size, 1,
+ padding=0, dilation=dilation,
+ groups=groups, bias=bias,
+ padding_mode=padding_mode,
+ device=device, dtype=dtype)
+ assert dilation == 1
+ self.causal_padding = kernel_size - 1
+ self.upsample = torch.nn.Upsample(scale_factor=stride, mode='nearest')
+
+ def forward(self, x: torch.Tensor, cache: torch.Tensor = torch.zeros(0, 0, 0)) -> Tuple[torch.Tensor, torch.Tensor]:
+ x = self.upsample(x)
+ input_timestep = x.shape[2]
+ if cache.size(2) == 0:
+ x = F.pad(x, (self.causal_padding, 0), value=0.0)
+ else:
+ assert cache.size(2) == self.causal_padding
+ x = torch.concat([cache, x], dim=2)
+ x = super(CausalConv1dUpsample, self).forward(x)
+ assert input_timestep == x.shape[2]
+ return x
diff --git a/freevideo_engine/dubbing/cosyvoice3/transformer/upsample_encoder.py b/freevideo_engine/dubbing/cosyvoice3/transformer/upsample_encoder.py
new file mode 100644
index 0000000..4067e8e
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/transformer/upsample_encoder.py
@@ -0,0 +1,60 @@
+# Copyright (c) 2021 Mobvoi Inc (Binbin Zhang, Di Wu)
+# 2022 Xingchen Song ([email protected])
+# 2024 Alibaba Inc (Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Modified from ESPnet(https://github.com/espnet/espnet)
+# Modified by FreeVideo: only PreLookaheadLayer is kept.
+import torch
+from torch import nn
+from torch.nn import functional as F
+
+
+class PreLookaheadLayer(nn.Module):
+ def __init__(self, in_channels: int, channels: int, pre_lookahead_len: int = 1):
+ super().__init__()
+ self.in_channels = in_channels
+ self.channels = channels
+ self.pre_lookahead_len = pre_lookahead_len
+ self.conv1 = nn.Conv1d(
+ in_channels, channels,
+ kernel_size=pre_lookahead_len + 1,
+ stride=1, padding=0,
+ )
+ self.conv2 = nn.Conv1d(
+ channels, in_channels,
+ kernel_size=3, stride=1, padding=0,
+ )
+
+ def forward(self, inputs: torch.Tensor, context: torch.Tensor = torch.zeros(0, 0, 0)) -> torch.Tensor:
+ """
+ inputs: (batch_size, seq_len, channels)
+ """
+ outputs = inputs.transpose(1, 2).contiguous()
+ context = context.transpose(1, 2).contiguous()
+ # look ahead
+ if context.size(2) == 0:
+ outputs = F.pad(outputs, (0, self.pre_lookahead_len), mode='constant', value=0.0)
+ else:
+ assert self.training is False, 'you have passed context, make sure that you are running inference mode'
+ assert context.size(2) == self.pre_lookahead_len
+ outputs = F.pad(torch.concat([outputs, context], dim=2), (0, self.pre_lookahead_len - context.size(2)), mode='constant', value=0.0)
+ outputs = F.leaky_relu(self.conv1(outputs))
+ # outputs
+ outputs = F.pad(outputs, (self.conv2.kernel_size[0] - 1, 0), mode='constant', value=0.0)
+ outputs = self.conv2(outputs)
+ outputs = outputs.transpose(1, 2).contiguous()
+
+ # residual connection
+ outputs = outputs + inputs
+ return outputs
diff --git a/freevideo_engine/dubbing/cosyvoice3/utils/__init__.py b/freevideo_engine/dubbing/cosyvoice3/utils/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/freevideo_engine/dubbing/cosyvoice3/utils/common.py b/freevideo_engine/dubbing/cosyvoice3/utils/common.py
new file mode 100644
index 0000000..3a8ad7a
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/utils/common.py
@@ -0,0 +1,167 @@
+# Copyright (c) 2020 Mobvoi Inc (Binbin Zhang)
+# 2024 Alibaba Inc (authors: Xiang Lyu)
+# 2025 Alibaba Inc (authors: Xiang Lyu, Bofan Zhou)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Modified from ESPnet(https://github.com/espnet/espnet)
+"""Unility functions for Transformer."""
+
+import random
+from typing import List
+
+import numpy as np
+import torch
+
+IGNORE_ID = -1
+
+def pad_list(xs: List[torch.Tensor], pad_value: int):
+ """Perform padding for the list of tensors.
+
+ Args:
+ xs (List): List of Tensors [(T_1, `*`), (T_2, `*`), ..., (T_B, `*`)].
+ pad_value (float): Value for padding.
+
+ Returns:
+ Tensor: Padded tensor (B, Tmax, `*`).
+
+ Examples:
+ >>> x = [torch.ones(4), torch.ones(2), torch.ones(1)]
+ >>> x
+ [tensor([1., 1., 1., 1.]), tensor([1., 1.]), tensor([1.])]
+ >>> pad_list(x, 0)
+ tensor([[1., 1., 1., 1.],
+ [1., 1., 0., 0.],
+ [1., 0., 0., 0.]])
+
+ """
+ max_len = max([len(item) for item in xs])
+ batchs = len(xs)
+ ndim = xs[0].ndim
+ if ndim == 1:
+ pad_res = torch.zeros(batchs,
+ max_len,
+ dtype=xs[0].dtype,
+ device=xs[0].device)
+ elif ndim == 2:
+ pad_res = torch.zeros(batchs,
+ max_len,
+ xs[0].shape[1],
+ dtype=xs[0].dtype,
+ device=xs[0].device)
+ elif ndim == 3:
+ pad_res = torch.zeros(batchs,
+ max_len,
+ xs[0].shape[1],
+ xs[0].shape[2],
+ dtype=xs[0].dtype,
+ device=xs[0].device)
+ else:
+ raise ValueError(f"Unsupported ndim: {ndim}")
+ pad_res.fill_(pad_value)
+ for i in range(batchs):
+ pad_res[i, :len(xs[i])] = xs[i]
+ return pad_res
+
+
+def th_accuracy(pad_outputs: torch.Tensor, pad_targets: torch.Tensor,
+ ignore_label: int) -> torch.Tensor:
+ """Calculate accuracy.
+
+ Args:
+ pad_outputs (Tensor): Prediction tensors (B * Lmax, D).
+ pad_targets (LongTensor): Target label tensors (B, Lmax).
+ ignore_label (int): Ignore label id.
+
+ Returns:
+ torch.Tensor: Accuracy value (0.0 - 1.0).
+
+ """
+ pad_pred = pad_outputs.view(pad_targets.size(0), pad_targets.size(1),
+ pad_outputs.size(1)).argmax(2)
+ mask = pad_targets != ignore_label
+ numerator = torch.sum(
+ pad_pred.masked_select(mask) == pad_targets.masked_select(mask))
+ denominator = torch.sum(mask)
+ return (numerator / denominator).detach()
+
+
+def get_padding(kernel_size, dilation=1):
+ return int((kernel_size * dilation - dilation) / 2)
+
+
+def init_weights(m, mean=0.0, std=0.01):
+ classname = m.__class__.__name__
+ if classname.find("Conv") != -1:
+ m.weight.data.normal_(mean, std)
+
+
+# Repetition Aware Sampling in VALL-E 2
+def ras_sampling(weighted_scores, decoded_tokens, sampling, top_p=0.8, top_k=25, win_size=10, tau_r=0.1):
+ top_ids = nucleus_sampling(weighted_scores, top_p=top_p, top_k=top_k)
+ rep_num = (torch.tensor(decoded_tokens[-win_size:]).to(weighted_scores.device) == top_ids).sum().item()
+ if rep_num >= win_size * tau_r:
+ weighted_scores[top_ids] = -float('inf')
+ top_ids = random_sampling(weighted_scores, decoded_tokens, sampling)
+ return top_ids
+
+
+def nucleus_sampling(weighted_scores, top_p=0.8, top_k=25):
+ prob, indices = [], []
+ cum_prob = 0.0
+ sorted_value, sorted_idx = weighted_scores.softmax(dim=0).sort(descending=True, stable=True)
+ for i in range(len(sorted_idx)):
+ # sampling both top-p and numbers.
+ if cum_prob < top_p and len(prob) < top_k:
+ cum_prob += sorted_value[i]
+ prob.append(sorted_value[i])
+ indices.append(sorted_idx[i])
+ else:
+ break
+ prob = torch.tensor(prob).to(weighted_scores)
+ indices = torch.tensor(indices, dtype=torch.long).to(weighted_scores.device)
+ top_ids = indices[prob.multinomial(1, replacement=True)].item()
+ return top_ids
+
+
+def random_sampling(weighted_scores, decoded_tokens, sampling):
+ top_ids = weighted_scores.softmax(dim=0).multinomial(1, replacement=True).item()
+ return top_ids
+
+
+def fade_in_out(fade_in_mel, fade_out_mel, window):
+ device = fade_in_mel.device
+ fade_in_mel, fade_out_mel = fade_in_mel.cpu(), fade_out_mel.cpu()
+ mel_overlap_len = int(window.shape[0] / 2)
+ if fade_in_mel.device == torch.device('cpu'):
+ fade_in_mel = fade_in_mel.clone()
+ fade_in_mel[..., :mel_overlap_len] = fade_in_mel[..., :mel_overlap_len] * window[:mel_overlap_len] + \
+ fade_out_mel[..., -mel_overlap_len:] * window[mel_overlap_len:]
+ return fade_in_mel.to(device)
+
+
+def set_all_random_seed(seed):
+ random.seed(seed)
+ np.random.seed(seed)
+ torch.manual_seed(seed)
+ torch.cuda.manual_seed_all(seed)
+
+
+def mask_to_bias(mask: torch.Tensor, dtype: torch.dtype) -> torch.Tensor:
+ assert mask.dtype == torch.bool
+ assert dtype in [torch.float32, torch.bfloat16, torch.float16]
+ mask = mask.to(dtype)
+ # attention mask bias
+ # NOTE(Mddct): torch.finfo jit issues
+ # chunk_masks = (1.0 - chunk_masks) * torch.finfo(dtype).min
+ mask = (1.0 - mask) * -1.0e+10
+ return mask
diff --git a/freevideo_engine/dubbing/cosyvoice3/utils/file_utils.py b/freevideo_engine/dubbing/cosyvoice3/utils/file_utils.py
new file mode 100644
index 0000000..e9a6e36
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/utils/file_utils.py
@@ -0,0 +1,32 @@
+# Copyright (c) 2021 Mobvoi Inc. (authors: Binbin Zhang)
+# 2024 Alibaba Inc (authors: Xiang Lyu, Zetao Hu)
+# 2025 Alibaba Inc (authors: Xiang Lyu, Yabin Li)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Modified by FreeVideo: only load_wav is kept.
+
+import logging
+
+import torch
+import torchaudio
+
+
+def load_wav(wav, target_sr, min_sr=16000):
+ from freevideo_engine.dubbing import wavio
+ data, sample_rate = wavio.read(wav)
+ speech = torch.from_numpy(data.T.copy())
+ speech = speech.mean(dim=0, keepdim=True)
+ if sample_rate != target_sr:
+ assert sample_rate >= min_sr, 'wav sample rate {} must be greater than {}'.format(sample_rate, target_sr)
+ speech = torchaudio.transforms.Resample(orig_freq=sample_rate, new_freq=target_sr)(speech)
+ return speech
diff --git a/freevideo_engine/dubbing/cosyvoice3/utils/frontend_utils.py b/freevideo_engine/dubbing/cosyvoice3/utils/frontend_utils.py
new file mode 100644
index 0000000..ea1c9fc
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/utils/frontend_utils.py
@@ -0,0 +1,136 @@
+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Zhihao Du)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import re
+import regex
+chinese_char_pattern = re.compile(r'[\u4e00-\u9fff]+')
+
+
+# whether contain chinese character
+def contains_chinese(text):
+ return bool(chinese_char_pattern.search(text))
+
+
+# replace special symbol
+def replace_corner_mark(text):
+ text = text.replace('²', '平方')
+ text = text.replace('³', '立方')
+ return text
+
+
+# remove meaningless symbol
+def remove_bracket(text):
+ text = text.replace('(', '').replace(')', '')
+ text = text.replace('【', '').replace('】', '')
+ text = text.replace('`', '').replace('`', '')
+ text = text.replace("——", " ")
+ return text
+
+
+# spell Arabic numerals
+def spell_out_number(text: str, inflect_parser):
+ new_text = []
+ st = None
+ for i, c in enumerate(text):
+ if not c.isdigit():
+ if st is not None:
+ num_str = inflect_parser.number_to_words(text[st: i])
+ new_text.append(num_str)
+ st = None
+ new_text.append(c)
+ else:
+ if st is None:
+ st = i
+ if st is not None and st < len(text):
+ num_str = inflect_parser.number_to_words(text[st:])
+ new_text.append(num_str)
+ return ''.join(new_text)
+
+
+# split paragrah logic:
+# 1. per sentence max len token_max_n, min len token_min_n, merge if last sentence len less than merge_len
+# 2. cal sentence len according to lang
+# 3. split sentence according to puncatation
+def split_paragraph(text: str, tokenize, lang="zh", token_max_n=80, token_min_n=60, merge_len=20, comma_split=False):
+ def calc_utt_length(_text: str):
+ if lang == "zh":
+ return len(_text)
+ else:
+ return len(tokenize(_text))
+
+ def should_merge(_text: str):
+ if lang == "zh":
+ return len(_text) < merge_len
+ else:
+ return len(tokenize(_text)) < merge_len
+
+ if lang == "zh":
+ pounc = ['。', '?', '!', ';', ':', '、', '.', '?', '!', ';']
+ else:
+ pounc = ['.', '?', '!', ';', ':']
+ if comma_split:
+ pounc.extend([',', ','])
+
+ if text[-1] not in pounc:
+ if lang == "zh":
+ text += "。"
+ else:
+ text += "."
+
+ st = 0
+ utts = []
+ for i, c in enumerate(text):
+ if c in pounc:
+ if len(text[st: i]) > 0:
+ utts.append(text[st: i] + c)
+ if i + 1 < len(text) and text[i + 1] in ['"', '”']:
+ tmp = utts.pop(-1)
+ utts.append(tmp + text[i + 1])
+ st = i + 2
+ else:
+ st = i + 1
+
+ final_utts = []
+ cur_utt = ""
+ for utt in utts:
+ if calc_utt_length(cur_utt + utt) > token_max_n and calc_utt_length(cur_utt) > token_min_n:
+ final_utts.append(cur_utt)
+ cur_utt = ""
+ cur_utt = cur_utt + utt
+ if len(cur_utt) > 0:
+ if should_merge(cur_utt) and len(final_utts) != 0:
+ final_utts[-1] = final_utts[-1] + cur_utt
+ else:
+ final_utts.append(cur_utt)
+
+ return final_utts
+
+
+# remove blank between chinese character
+def replace_blank(text: str):
+ out_str = []
+ for i, c in enumerate(text):
+ if c == " ":
+ if ((text[i + 1].isascii() and text[i + 1] != " ") and
+ (text[i - 1].isascii() and text[i - 1] != " ")):
+ out_str.append(c)
+ else:
+ out_str.append(c)
+ return "".join(out_str)
+
+
+def is_only_punctuation(text):
+ # Regular expression: Match strings that consist only of punctuation marks or are empty.
+ punctuation_pattern = r'^[\p{P}\p{S}]*$'
+ return bool(regex.fullmatch(punctuation_pattern, text))
diff --git a/freevideo_engine/dubbing/cosyvoice3/utils/mask.py b/freevideo_engine/dubbing/cosyvoice3/utils/mask.py
new file mode 100644
index 0000000..5d3dfd6
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/utils/mask.py
@@ -0,0 +1,265 @@
+# Copyright (c) 2019 Shigeki Karita
+# 2020 Mobvoi Inc (Binbin Zhang)
+# 2024 Alibaba Inc (authors: Xiang Lyu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import torch
+'''
+def subsequent_mask(
+ size: int,
+ device: torch.device = torch.device("cpu"),
+) -> torch.Tensor:
+ """Create mask for subsequent steps (size, size).
+
+ This mask is used only in decoder which works in an auto-regressive mode.
+ This means the current step could only do attention with its left steps.
+
+ In encoder, fully attention is used when streaming is not necessary and
+ the sequence is not long. In this case, no attention mask is needed.
+
+ When streaming is need, chunk-based attention is used in encoder. See
+ subsequent_chunk_mask for the chunk-based attention mask.
+
+ Args:
+ size (int): size of mask
+ str device (str): "cpu" or "cuda" or torch.Tensor.device
+ dtype (torch.device): result dtype
+
+ Returns:
+ torch.Tensor: mask
+
+ Examples:
+ >>> subsequent_mask(3)
+ [[1, 0, 0],
+ [1, 1, 0],
+ [1, 1, 1]]
+ """
+ ret = torch.ones(size, size, device=device, dtype=torch.bool)
+ return torch.tril(ret)
+'''
+
+
+def subsequent_mask(
+ size: int,
+ device: torch.device = torch.device("cpu"),
+) -> torch.Tensor:
+ """Create mask for subsequent steps (size, size).
+
+ This mask is used only in decoder which works in an auto-regressive mode.
+ This means the current step could only do attention with its left steps.
+
+ In encoder, fully attention is used when streaming is not necessary and
+ the sequence is not long. In this case, no attention mask is needed.
+
+ When streaming is need, chunk-based attention is used in encoder. See
+ subsequent_chunk_mask for the chunk-based attention mask.
+
+ Args:
+ size (int): size of mask
+ str device (str): "cpu" or "cuda" or torch.Tensor.device
+ dtype (torch.device): result dtype
+
+ Returns:
+ torch.Tensor: mask
+
+ Examples:
+ >>> subsequent_mask(3)
+ [[1, 0, 0],
+ [1, 1, 0],
+ [1, 1, 1]]
+ """
+ arange = torch.arange(size, device=device)
+ mask = arange.expand(size, size)
+ arange = arange.unsqueeze(-1)
+ mask = mask <= arange
+ return mask
+
+
+def subsequent_chunk_mask_deprecated(
+ size: int,
+ chunk_size: int,
+ num_left_chunks: int = -1,
+ device: torch.device = torch.device("cpu"),
+) -> torch.Tensor:
+ """Create mask for subsequent steps (size, size) with chunk size,
+ this is for streaming encoder
+
+ Args:
+ size (int): size of mask
+ chunk_size (int): size of chunk
+ num_left_chunks (int): number of left chunks
+ <0: use full chunk
+ >=0: use num_left_chunks
+ device (torch.device): "cpu" or "cuda" or torch.Tensor.device
+
+ Returns:
+ torch.Tensor: mask
+
+ Examples:
+ >>> subsequent_chunk_mask(4, 2)
+ [[1, 1, 0, 0],
+ [1, 1, 0, 0],
+ [1, 1, 1, 1],
+ [1, 1, 1, 1]]
+ """
+ ret = torch.zeros(size, size, device=device, dtype=torch.bool)
+ for i in range(size):
+ if num_left_chunks < 0:
+ start = 0
+ else:
+ start = max((i // chunk_size - num_left_chunks) * chunk_size, 0)
+ ending = min((i // chunk_size + 1) * chunk_size, size)
+ ret[i, start:ending] = True
+ return ret
+
+
+def subsequent_chunk_mask(
+ size: int,
+ chunk_size: int,
+ num_left_chunks: int = -1,
+ device: torch.device = torch.device("cpu"),
+) -> torch.Tensor:
+ """Create mask for subsequent steps (size, size) with chunk size,
+ this is for streaming encoder
+
+ Args:
+ size (int): size of mask
+ chunk_size (int): size of chunk
+ num_left_chunks (int): number of left chunks
+ <0: use full chunk
+ >=0: use num_left_chunks
+ device (torch.device): "cpu" or "cuda" or torch.Tensor.device
+
+ Returns:
+ torch.Tensor: mask
+
+ Examples:
+ >>> subsequent_chunk_mask(4, 2)
+ [[1, 1, 0, 0],
+ [1, 1, 0, 0],
+ [1, 1, 1, 1],
+ [1, 1, 1, 1]]
+ """
+ # NOTE this modified implementation meets onnx export requirements, but it doesn't support num_left_chunks
+ pos_idx = torch.arange(size, device=device)
+ block_value = (torch.div(pos_idx, chunk_size, rounding_mode='trunc') + 1) * chunk_size
+ ret = pos_idx.unsqueeze(0) < block_value.unsqueeze(1)
+ return ret
+
+
+def add_optional_chunk_mask(xs: torch.Tensor,
+ masks: torch.Tensor,
+ use_dynamic_chunk: bool,
+ use_dynamic_left_chunk: bool,
+ decoding_chunk_size: int,
+ static_chunk_size: int,
+ num_decoding_left_chunks: int,
+ enable_full_context: bool = True):
+ """ Apply optional mask for encoder.
+
+ Args:
+ xs (torch.Tensor): padded input, (B, L, D), L for max length
+ mask (torch.Tensor): mask for xs, (B, 1, L)
+ use_dynamic_chunk (bool): whether to use dynamic chunk or not
+ use_dynamic_left_chunk (bool): whether to use dynamic left chunk for
+ training.
+ decoding_chunk_size (int): decoding chunk size for dynamic chunk, it's
+ 0: default for training, use random dynamic chunk.
+ <0: for decoding, use full chunk.
+ >0: for decoding, use fixed chunk size as set.
+ static_chunk_size (int): chunk size for static chunk training/decoding
+ if it's greater than 0, if use_dynamic_chunk is true,
+ this parameter will be ignored
+ num_decoding_left_chunks: number of left chunks, this is for decoding,
+ the chunk size is decoding_chunk_size.
+ >=0: use num_decoding_left_chunks
+ <0: use all left chunks
+ enable_full_context (bool):
+ True: chunk size is either [1, 25] or full context(max_len)
+ False: chunk size ~ U[1, 25]
+
+ Returns:
+ torch.Tensor: chunk mask of the input xs.
+ """
+ # Whether to use chunk mask or not
+ if use_dynamic_chunk:
+ max_len = xs.size(1)
+ if decoding_chunk_size < 0:
+ chunk_size = max_len
+ num_left_chunks = -1
+ elif decoding_chunk_size > 0:
+ chunk_size = decoding_chunk_size
+ num_left_chunks = num_decoding_left_chunks
+ else:
+ # chunk size is either [1, 25] or full context(max_len).
+ # Since we use 4 times subsampling and allow up to 1s(100 frames)
+ # delay, the maximum frame is 100 / 4 = 25.
+ chunk_size = torch.randint(1, max_len, (1, )).item()
+ num_left_chunks = -1
+ if chunk_size > max_len // 2 and enable_full_context:
+ chunk_size = max_len
+ else:
+ chunk_size = chunk_size % 25 + 1
+ if use_dynamic_left_chunk:
+ max_left_chunks = (max_len - 1) // chunk_size
+ num_left_chunks = torch.randint(0, max_left_chunks,
+ (1, )).item()
+ chunk_masks = subsequent_chunk_mask(xs.size(1), chunk_size,
+ num_left_chunks,
+ xs.device) # (L, L)
+ chunk_masks = chunk_masks.unsqueeze(0) # (1, L, L)
+ chunk_masks = masks & chunk_masks # (B, L, L)
+ elif static_chunk_size > 0:
+ num_left_chunks = num_decoding_left_chunks
+ chunk_masks = subsequent_chunk_mask(xs.size(1), static_chunk_size,
+ num_left_chunks,
+ xs.device) # (L, L)
+ chunk_masks = chunk_masks.unsqueeze(0) # (1, L, L)
+ chunk_masks = masks & chunk_masks # (B, L, L)
+ else:
+ chunk_masks = masks
+ assert chunk_masks.dtype == torch.bool
+ if (chunk_masks.sum(dim=-1) == 0).sum().item() != 0:
+ print('get chunk_masks all false at some timestep, force set to true, make sure they are masked in futuer computation!')
+ chunk_masks[chunk_masks.sum(dim=-1) == 0] = True
+ return chunk_masks
+
+
+def make_pad_mask(lengths: torch.Tensor, max_len: int = 0) -> torch.Tensor:
+ """Make mask tensor containing indices of padded part.
+
+ See description of make_non_pad_mask.
+
+ Args:
+ lengths (torch.Tensor): Batch of lengths (B,).
+ Returns:
+ torch.Tensor: Mask tensor containing indices of padded part.
+
+ Examples:
+ >>> lengths = [5, 3, 2]
+ >>> make_pad_mask(lengths)
+ masks = [[0, 0, 0, 0 ,0],
+ [0, 0, 0, 1, 1],
+ [0, 0, 1, 1, 1]]
+ """
+ batch_size = lengths.size(0)
+ max_len = max_len if max_len > 0 else lengths.max().item()
+ seq_range = torch.arange(0,
+ max_len,
+ dtype=torch.int64,
+ device=lengths.device)
+ seq_range_expand = seq_range.unsqueeze(0).expand(batch_size, max_len)
+ seq_length_expand = lengths.unsqueeze(-1)
+ mask = seq_range_expand >= seq_length_expand
+ return mask
diff --git a/freevideo_engine/dubbing/cosyvoice3/utils/mel.py b/freevideo_engine/dubbing/cosyvoice3/utils/mel.py
new file mode 100644
index 0000000..bca293d
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3/utils/mel.py
@@ -0,0 +1,98 @@
+import numpy as np
+import torch
+
+
+def _hz_to_mel(f):
+ f = np.asarray(f, dtype=np.float64)
+ lin = f / (200.0 / 3)
+ log = 15.0 + np.log(np.maximum(f, 1e-10) / 1000.0) / (np.log(6.4) / 27.0)
+ return np.where(f >= 1000.0, log, lin)
+
+
+def _mel_to_hz(m):
+ m = np.asarray(m, dtype=np.float64)
+ lin = (200.0 / 3) * m
+ log = 1000.0 * np.exp((np.log(6.4) / 27.0) * (m - 15.0))
+ return np.where(m >= 15.0, log, lin)
+
+
+def librosa_mel_fn(sr, n_fft, n_mels, fmin=0.0, fmax=None):
+ """Slaney-scale, Slaney-normalised mel filterbank (same result as librosa.filters.mel, float32)."""
+ fmax = sr / 2.0 if fmax is None else fmax
+ freqs = np.linspace(0.0, sr / 2.0, n_fft // 2 + 1)
+ pts = _mel_to_hz(np.linspace(_hz_to_mel(fmin), _hz_to_mel(fmax), n_mels + 2))
+ fdiff = np.diff(pts)
+ ramps = pts[:, None] - freqs[None, :]
+ lower = -ramps[:-2] / fdiff[:-1, None]
+ upper = ramps[2:] / fdiff[1:, None]
+ w = np.maximum(0.0, np.minimum(lower, upper))
+ w *= (2.0 / (pts[2:n_mels + 2] - pts[:n_mels]))[:, None]
+ return w.astype(np.float32)
+
+_FILTERS = {}
+
+
+def log_mel_spectrogram(audio, n_mels=128, n_fft=400, hop=160, sr=16000):
+ """Whisper-compatible log-mel spectrogram (Slaney filters, log10, floor at max - 8, scaled to ~[-1, 1])."""
+ key = (n_mels, n_fft, sr, audio.device)
+ if key not in _FILTERS:
+ _FILTERS[key] = torch.from_numpy(librosa_mel_fn(sr, n_fft, n_mels)).to(audio.device)
+ window = torch.hann_window(n_fft).to(audio.device)
+ stft = torch.stft(audio, n_fft, hop, window=window, return_complex=True)
+ magnitudes = stft[..., :-1].abs() ** 2
+ log_spec = torch.clamp(_FILTERS[key] @ magnitudes, min=1e-10).log10()
+ log_spec = torch.maximum(log_spec, log_spec.max() - 8.0)
+ return (log_spec + 4.0) / 4.0
+
+
+# mel_spectrogram below is copied from Matcha-TTS matcha/utils/audio.py (MIT; derived from HiFi-GAN, MIT)
+def dynamic_range_compression_torch(x, C=1, clip_val=1e-5):
+ return torch.log(torch.clamp(x, min=clip_val) * C)
+
+
+def spectral_normalize_torch(magnitudes):
+ return dynamic_range_compression_torch(magnitudes)
+
+
+mel_basis = {}
+hann_window = {}
+
+
+def mel_spectrogram(y, n_fft, num_mels, sampling_rate, hop_size, win_size, fmin, fmax, center=False):
+ if torch.min(y) < -1.0:
+ print("min value is ", torch.min(y))
+ if torch.max(y) > 1.0:
+ print("max value is ", torch.max(y))
+
+ global mel_basis, hann_window # pylint: disable=global-statement,global-variable-not-assigned
+ if f"{str(fmax)}_{str(y.device)}" not in mel_basis:
+ mel = librosa_mel_fn(sr=sampling_rate, n_fft=n_fft, n_mels=num_mels, fmin=fmin, fmax=fmax)
+ mel_basis[str(fmax) + "_" + str(y.device)] = torch.from_numpy(mel).float().to(y.device)
+ hann_window[str(y.device)] = torch.hann_window(win_size).to(y.device)
+
+ y = torch.nn.functional.pad(
+ y.unsqueeze(1), (int((n_fft - hop_size) / 2), int((n_fft - hop_size) / 2)), mode="reflect"
+ )
+ y = y.squeeze(1)
+
+ spec = torch.view_as_real(
+ torch.stft(
+ y,
+ n_fft,
+ hop_length=hop_size,
+ win_length=win_size,
+ window=hann_window[str(y.device)],
+ center=center,
+ pad_mode="reflect",
+ normalized=False,
+ onesided=True,
+ return_complex=True,
+ )
+ )
+
+ spec = torch.sqrt(spec.pow(2).sum(-1) + (1e-9))
+
+ spec = torch.matmul(mel_basis[str(fmax) + "_" + str(y.device)], spec)
+ spec = spectral_normalize_torch(spec)
+
+ return spec
diff --git a/freevideo_engine/dubbing/cosyvoice3_pack.json b/freevideo_engine/dubbing/cosyvoice3_pack.json
new file mode 100644
index 0000000..9becf95
--- /dev/null
+++ b/freevideo_engine/dubbing/cosyvoice3_pack.json
@@ -0,0 +1,159 @@
+{
+ "format": 1,
+ "id": "cosyvoice3",
+ "name": "Fun-CosyVoice3-0.5B-2512",
+ "license": "Apache-2.0",
+ "source": "https://github.com/FunAudioLLM/CosyVoice",
+ "modelscope": {
+ "repo": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
+ "revision": "master"
+ },
+ "huggingface": {
+ "repo": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512"
+ },
+ "files": [
+ {
+ "file": "llm.pt",
+ "bytes": 2024669519,
+ "sha256": "69f43bd545131c30e98947fb360ea8b4dc9916d8e83dded7757c7ea4f5a24970"
+ },
+ {
+ "file": "flow.pt",
+ "bytes": 1329116148,
+ "sha256": "a6fab32a7825e5b0bc855ddd948f8db9370b0a786fbc249caa4595e95b608e4b"
+ },
+ {
+ "file": "hift.pt",
+ "bytes": 83202622,
+ "sha256": "b279d7641eb97ae55b3b540cfba4f953c26492a2df758328a89a4d007ab87a65"
+ },
+ {
+ "file": "campplus.onnx",
+ "bytes": 28303423,
+ "sha256": "a6ac6a63997761ae2997373e2ee1c47040854b4b759ea41ec48e4e42df0f4d73"
+ },
+ {
+ "file": "speech_tokenizer_v3.onnx",
+ "bytes": 969451503,
+ "sha256": "23236a74175dbdda47afc66dbadd5bcb41303c467a57c261cb8539ad9db9208d"
+ },
+ {
+ "file": "CosyVoice-BlankEN/config.json",
+ "bytes": 659,
+ "sha256": "168aa1bd401abc3bc262ba15ba4e499627a8b4e006e9d050b47c22de20660185"
+ },
+ {
+ "file": "CosyVoice-BlankEN/generation_config.json",
+ "bytes": 242,
+ "sha256": "e558847a8b4402616f1273797b015104dc266fe4b520056fca88823ba8f8ebe6"
+ },
+ {
+ "file": "CosyVoice-BlankEN/merges.txt",
+ "bytes": 1402109,
+ "sha256": "ac8ff86a72bee70828fbc1119bc4398c6f3a9a6e490d7b0dbe917be025478bd0"
+ },
+ {
+ "file": "CosyVoice-BlankEN/tokenizer_config.json",
+ "bytes": 1287,
+ "sha256": "482bd979881423375ca5414e4e0d94cd7c5349dbb17fffd46b4d36d71e62a1bc"
+ },
+ {
+ "file": "CosyVoice-BlankEN/vocab.json",
+ "bytes": 2776833,
+ "sha256": "ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910"
+ }
+ ],
+ "runtime": {
+ "python": "cp312",
+ "note": "Hash-locked PyPI wheels, downloaded when the voice pack is installed; not part of the installer.",
+ "wheels": {
+ "linux-x86_64": [
+ {
+ "name": "onnxruntime",
+ "version": "1.31.1",
+ "license": "MIT",
+ "file": "onnxruntime-1.31.1-cp312-cp312-manylinux_2_28_x86_64.whl",
+ "bytes": 23760988,
+ "sha256": "1f3dea47f2f0e5117c2404cea7d21fd253ae7836a3bf167a501cdfbb2600d536",
+ "path": "eb/27/c6a823bac55b2977bea2027c3ec9393a2c1002cb9a09eb40354563e02730/onnxruntime-1.31.1-cp312-cp312-manylinux_2_28_x86_64.whl"
+ },
+ {
+ "name": "flatbuffers",
+ "version": "25.12.19",
+ "license": "Apache-2.0",
+ "file": "flatbuffers-25.12.19-py2.py3-none-any.whl",
+ "bytes": 26661,
+ "sha256": "7634f50c427838bb021c2d66a3d1168e9d199b0607e6329399f04846d42e20b4",
+ "path": "e8/2d/d2a548598be01649e2d46231d151a6c56d10b964d94043a335ae56ea2d92/flatbuffers-25.12.19-py2.py3-none-any.whl"
+ },
+ {
+ "name": "protobuf",
+ "version": "7.36.2",
+ "license": "BSD-3-Clause",
+ "file": "protobuf-7.36.2-cp310-abi3-manylinux2014_x86_64.whl",
+ "bytes": 343223,
+ "sha256": "89f23aa53c24553a2416fd4fd1ec06f74fa42b14b546d8883128813f775bbfd2",
+ "path": "db/f3/3996583dd2906297a637af12114deddf7658af6e683fedb83be061983fb5/protobuf-7.36.2-cp310-abi3-manylinux2014_x86_64.whl"
+ }
+ ],
+ "windows-x64": [
+ {
+ "name": "onnxruntime",
+ "version": "1.31.1",
+ "license": "MIT",
+ "file": "onnxruntime-1.31.1-cp312-cp312-win_amd64.whl",
+ "bytes": 14888466,
+ "sha256": "ae8a9a351efda685cc59a66226cdefaa37a126c1d66872ca23a2de9425326130",
+ "path": "45/b2/793b4051d8dc6acb379ead9b72050f0ffa283ca49a6904add2e218b83e2e/onnxruntime-1.31.1-cp312-cp312-win_amd64.whl"
+ },
+ {
+ "name": "flatbuffers",
+ "version": "25.12.19",
+ "license": "Apache-2.0",
+ "file": "flatbuffers-25.12.19-py2.py3-none-any.whl",
+ "bytes": 26661,
+ "sha256": "7634f50c427838bb021c2d66a3d1168e9d199b0607e6329399f04846d42e20b4",
+ "path": "e8/2d/d2a548598be01649e2d46231d151a6c56d10b964d94043a335ae56ea2d92/flatbuffers-25.12.19-py2.py3-none-any.whl"
+ },
+ {
+ "name": "protobuf",
+ "version": "7.36.2",
+ "license": "BSD-3-Clause",
+ "file": "protobuf-7.36.2-cp310-abi3-win_amd64.whl",
+ "bytes": 456514,
+ "sha256": "a300819d441e078a5608c0d3c709796bb548136058fda017ae51d425b44fd353",
+ "path": "8a/55/b77bda4e5e5f5971fb51b07663694690e9afdb9402136c16a522bd621cad/protobuf-7.36.2-cp310-abi3-win_amd64.whl"
+ }
+ ],
+ "macos-arm64": [
+ {
+ "name": "onnxruntime",
+ "version": "1.31.1",
+ "license": "MIT",
+ "file": "onnxruntime-1.31.1-cp312-cp312-macosx_14_0_arm64.whl",
+ "bytes": 20882125,
+ "sha256": "96dd2564b10ea7153ad9c207e7cd8e5917cb3ba521e0e0a62ac03486fdc8d56c",
+ "path": "0e/f7/cecc0efeeea83bff2f7789ae1b80f27fa0f0210d6d9d66d1f0031a6b5e47/onnxruntime-1.31.1-cp312-cp312-macosx_14_0_arm64.whl"
+ },
+ {
+ "name": "flatbuffers",
+ "version": "25.12.19",
+ "license": "Apache-2.0",
+ "file": "flatbuffers-25.12.19-py2.py3-none-any.whl",
+ "bytes": 26661,
+ "sha256": "7634f50c427838bb021c2d66a3d1168e9d199b0607e6329399f04846d42e20b4",
+ "path": "e8/2d/d2a548598be01649e2d46231d151a6c56d10b964d94043a335ae56ea2d92/flatbuffers-25.12.19-py2.py3-none-any.whl"
+ },
+ {
+ "name": "protobuf",
+ "version": "7.36.2",
+ "license": "BSD-3-Clause",
+ "file": "protobuf-7.36.2-cp310-abi3-macosx_10_9_universal2.whl",
+ "bytes": 456039,
+ "sha256": "cbc70b17ee27e28894c7fee8bb04be1abead49e936bc70eb60052531eee2079e",
+ "path": "32/72/98342feb672507c8f3a69e34b4fa8961f608edba5c1a48a6f47156d92cb5/protobuf-7.36.2-cp310-abi3-macosx_10_9_universal2.whl"
+ }
+ ]
+ }
+ }
+}
diff --git a/freevideo_engine/dubbing/pack.py b/freevideo_engine/dubbing/pack.py
new file mode 100644
index 0000000..5154869
--- /dev/null
+++ b/freevideo_engine/dubbing/pack.py
@@ -0,0 +1,365 @@
+"""On-demand download of the dubbing model (Fun-CosyVoice3-0.5B-2512) and its inference runtime.
+
+Nothing here ships in the installer. The catalog (`cosyvoice3_pack.json`) lists every file with its
+size and sha256; a file is used only after it matched. Layout under `<model_root>/dubbing/`:
+`cosyvoice3/` (the weights), `wheels/` (the runtime wheels as downloaded), `runtime/<id>/` (the wheels
+unpacked; put on `sys.path` before `import onnxruntime`) and `licenses/` (the runtime licence texts).
+ModelScope has no commit that matches a Hugging Face one, so every source is accepted only when the
+size and SHA-256 in the catalog match, and the file names are the same on all of them. The runtime
+wheels are the PyPI files, fetched from PyPI or its domestic mirror by the same rule. onnxruntime also
+requires packaging and sympy; both come with torch in the unified environment, so they are not listed.
+The wheels are chosen for the Python of the service process, which is the pinned 3.12 of the product's
+own environment. A user's own ComfyUI Python of another version would fail the import in the worker,
+which reports it as a general failure.
+"""
+import hashlib
+import json
+import os
+from pathlib import Path
+import shutil
+import sys
+import zipfile
+
+from .. import disk_space
+
+FAMILY = 'image-models'
+RESERVE = 256 * 2**20
+LEDGER = 'download-verified.json'
+NAME = 'cosyvoice3'
+COMPLETE = 'complete.json'
+MAX_UNPACKED = 256 * 2**20
+LICENSES = 'RUNTIME-LICENSES.txt'
+# The two PyPI hosts of the `pypi` download family; a wheel's `path` is the same on both.
+PYPI_FILES = {'official': 'https://files.pythonhosted.org/packages/', 'tuna': 'https://pypi.tuna.tsinghua.edu.cn/packages/'}
+MODULES = (('onnxruntime', '__init__.py'), ('flatbuffers', '__init__.py'), ('google', 'protobuf', '__init__.py'))
+
+
+class PackError(Exception):
+ def __init__(self, reason):
+ super().__init__(reason)
+ self.reason = reason
+
+
+def catalog():
+ """The file list shipped beside this file; a frozen launcher's bundled copy answers when it is gone."""
+ name = 'cosyvoice3_pack.json'
+ paths = [Path(__file__).with_name(name)]
+ bundle = getattr(sys, '_MEIPASS', None)
+ if bundle:
+ paths.append(Path(bundle) / 'engine-source' / 'freevideo_engine' / 'dubbing' / name)
+ for path in paths:
+ try:
+ return json.loads(path.read_text(encoding='utf-8'))
+ except (OSError, ValueError) as error:
+ missing = error
+ raise missing
+
+
+def rows(spec=None):
+ return list((spec or catalog())['files'])
+
+
+def base(root=None):
+ if root is None:
+ from ..paths import model_root
+ root = model_root()
+ return Path(root) / 'dubbing'
+
+
+def directory(root=None):
+ return base(root) / NAME
+
+
+def _ledger(folder):
+ try:
+ value = json.loads((folder / LEDGER).read_text(encoding='utf-8'))
+ return value if isinstance(value, dict) else {}
+ except (OSError, ValueError):
+ return {}
+
+
+def _intact(folder, row, ledger):
+ """Size and the stamp recorded after the last hash match; no hashing here."""
+ from .. import provision
+ path = folder / row['file']
+ try:
+ if not path.is_file() or path.stat().st_size != row['bytes']:
+ return False
+ prior = ledger.get(str(path))
+ return isinstance(prior, dict) and all(
+ prior.get(key) == value for key, value in provision.file_identity(path, row).items())
+ except OSError:
+ return False
+
+
+def wheel_rows(spec=None):
+ """This machine's runtime wheels, or None (other platform or Python, or none listed)."""
+ from ..subtitle_pack import platform_key
+ runtime = (spec or catalog()).get('runtime') or {}
+ if runtime.get('python') != 'cp%d%d' % sys.version_info[:2]:
+ return None
+ return list((runtime.get('wheels') or {}).get(platform_key()) or []) or None
+
+
+def runtime_id(wheels):
+ return hashlib.sha256('\n'.join(row['sha256'] for row in wheels).encode('ascii')).hexdigest()[:12]
+
+
+def runtime_folder(root=None, spec=None):
+ wheels = wheel_rows(spec)
+ return base(root) / 'runtime' / runtime_id(wheels) if wheels else None
+
+
+def _unpacked(root=None, spec=None):
+ folder = runtime_folder(root, spec)
+ if folder is None:
+ return False
+ try:
+ receipt = json.loads((folder / COMPLETE).read_text(encoding='utf-8'))
+ return receipt.get('id') == folder.name and all(folder.joinpath(*parts).is_file() for parts in MODULES)
+ except (OSError, ValueError, KeyError, TypeError):
+ return False
+
+
+def runtime_path(root=None):
+ """The folder to put on sys.path for onnxruntime, or None while it is not unpacked."""
+ try:
+ return str(runtime_folder(root)) if _unpacked(root) else None
+ except (OSError, ValueError, KeyError, TypeError):
+ return None
+
+
+def activate(path):
+ """Put an unpacked runtime on sys.path (once); Windows also needs its DLL folders."""
+ if not path:
+ return False
+ folder = Path(path)
+ if str(folder) not in sys.path:
+ sys.path.insert(0, str(folder))
+ if sys.platform == 'win32' and hasattr(os, 'add_dll_directory'):
+ for item in {folder} | {found.parent for found in folder.rglob('*.dll')}:
+ os.add_dll_directory(str(item))
+ return True
+
+
+def ready(root=None, spec=None):
+ """Cheap and safe on a UI thread: sizes and the identity stamps of the last content check."""
+ try:
+ spec = spec or catalog()
+ folder = directory(root)
+ ledger = _ledger(folder)
+ return _unpacked(root, spec) and all(_intact(folder, row, ledger) for row in rows(spec))
+ except (OSError, ValueError, KeyError, TypeError):
+ return False
+
+
+def settle(root=None, spec=None):
+ """Same answer, but a file that moved without a stamp (a copied folder) is hashed here once.
+ For request handlers, never for a UI thread or under a lock."""
+ if ready(root, spec):
+ return True
+ try:
+ from .. import provision
+ from ..monitoring import save
+ spec = spec or catalog()
+ folder = directory(root)
+ if not _unpacked(root, spec) or not folder.is_dir():
+ return False
+ ledger = _ledger(folder)
+ found = all(provision.verified(folder / row['file'], row, ledger) for row in rows(spec))
+ if found:
+ save(folder / LEDGER, ledger)
+ return found
+ except (OSError, ValueError, KeyError, TypeError):
+ return False
+
+
+def missing_bytes(root=None, spec=None):
+ spec = spec or catalog()
+ folder = directory(root)
+ ledger = _ledger(folder)
+ weights = sum(row['bytes'] for row in rows(spec) if not _intact(folder, row, ledger))
+ return weights + (0 if _unpacked(root, spec) else sum(row['bytes'] for row in wheel_rows(spec) or ()))
+
+
+def status(root=None):
+ """What a page may show: the size of the download and whether the model is in place."""
+ spec = catalog()
+ wheels = wheel_rows(spec)
+ total = sum(row['bytes'] for row in rows(spec)) + sum(row['bytes'] for row in wheels or ())
+ return dict(name=NAME, license=spec.get('license'), bytes=missing_bytes(root, spec), total=total,
+ ready=ready(root, spec), supported=wheels is not None,
+ published=bool(spec.get('modelscope') or spec.get('huggingface')))
+
+
+def remote(spec, row):
+ """One catalog file with every source FreeVideo may fetch it from (ModelScope, Hugging Face, mirror)."""
+ from ..network import source_url
+ sources = []
+ mirror = spec.get('modelscope')
+ if mirror:
+ sources.append(dict(id='modelscope', url=source_url(FAMILY, 'modelscope') + '/models/%s/resolve/%s/%s' % (
+ mirror['repo'], mirror.get('revision', 'master'), row['file'])))
+ hub = spec.get('huggingface')
+ if hub:
+ path = '/%s/resolve/%s/%s' % (hub['repo'], hub.get('revision', 'main'), row['file'])
+ sources.append(dict(id='official', url=source_url('models', 'official') + path))
+ sources.append(dict(id='hf-mirror', url=source_url('models', 'hf-mirror') + path))
+ return dict(repo=(mirror or hub)['repo'], revision=(mirror or hub).get('revision', 'main'), file=row['file'],
+ family='image', sources=sources)
+
+
+def runtime_plan(plan):
+ """The plan for the runtime wheels: the `image-models` list holds the PyPI hosts in the order the
+ plan prefers them (PyPI, or its domestic mirror)."""
+ from ..network import ordered
+ names = [name for name in ordered(plan, 'pypi') if name in PYPI_FILES] or ['official']
+ return dict(plan, sources=dict(plan.get('sources', {}), **{FAMILY: [{'id': name} for name in names]}))
+
+
+def runtime_remote(row):
+ return dict(repo='pypi/' + row['name'], revision=row['version'], file=row['file'], family='image',
+ sources=[dict(id=name, url=PYPI_FILES[name] + row['path']) for name in PYPI_FILES])
+
+
+def _unpack(root, wheels, check):
+ """Extract the verified wheels into runtime/<id>/ and drop older runtimes. A file that two wheels
+ both carry, an unsafe name, or too much data fails the whole unpack."""
+ folder = runtime_folder(root)
+ parent = folder.parent
+ parent.mkdir(parents=True, exist_ok=True)
+ work = parent / ('.unpack-' + folder.name)
+ shutil.rmtree(work, ignore_errors=True)
+ total = 0
+ try:
+ for row in wheels:
+ with zipfile.ZipFile(wheels_folder(root) / row['file']) as archive:
+ for item in archive.infolist():
+ check()
+ name = item.filename
+ parts = Path(name).parts
+ if (name.startswith('/') or '\\' in name or ':' in name or not parts or '..' in parts
+ or (item.external_attr >> 16) & 0o170000 == 0o120000):
+ raise PackError('failed')
+ total += item.file_size
+ if total > MAX_UNPACKED:
+ raise PackError('failed')
+ target = work.joinpath(*parts)
+ if item.is_dir():
+ target.mkdir(parents=True, exist_ok=True)
+ continue
+ if target.exists():
+ raise PackError('failed')
+ target.parent.mkdir(parents=True, exist_ok=True)
+ with archive.open(item) as source, target.open('wb') as stream:
+ shutil.copyfileobj(source, stream, 1024 * 1024)
+ if not all(work.joinpath(*parts).is_file() for parts in MODULES):
+ raise PackError('failed')
+ write_licenses(root, work, wheels)
+ (work / COMPLETE).write_text(json.dumps(dict(id=folder.name, bytes=total)), encoding='utf-8')
+ shutil.rmtree(folder, ignore_errors=True)
+ os.replace(work, folder)
+ finally:
+ shutil.rmtree(work, ignore_errors=True)
+ for other in parent.iterdir():
+ if other.is_dir() and other != folder and not other.name.startswith('.unpack-'):
+ shutil.rmtree(other, ignore_errors=True)
+
+
+def write_licenses(root, unpacked, wheels):
+ """The runtime's licence texts, with the pack and not with the installer: the name, version and
+ licence of every wheel, then the licence files its metadata carries."""
+ parts = ['Runtime libraries of the dubbing model pack. They are downloaded when the pack is installed.\n']
+ for row in wheels:
+ parts.append('\n%s\n%s %s, %s\n%s\n' % ('=' * 72, row['name'], row['version'], row['license'], '=' * 72))
+ for info in sorted(unpacked.glob('%s-%s.dist-info' % (row['name'], row['version']))):
+ for path in sorted(info.rglob('*')):
+ if path.is_file() and path.suffix in ('', '.txt', '.md') and any(
+ word in path.name.upper() for word in ('LICENSE', 'NOTICE', 'COPYING')):
+ parts.append('\n--- %s ---\n%s\n' % (path.name, path.read_text(encoding='utf-8', errors='replace')))
+ folder = base(root) / 'licenses'
+ folder.mkdir(parents=True, exist_ok=True)
+ (folder / LICENSES).write_text(''.join(parts), encoding='utf-8')
+
+
+def wheels_folder(root=None):
+ return base(root) / 'wheels'
+
+
+def install(root=None, progress=None, check=None):
+ """Download and verify the missing runtime wheels and model files, then unpack the runtime.
+ Resumable; a file is kept only after its hash matched."""
+ from .. import network, provision
+ from ..adaln_assets import _download_plan
+ from ..monitoring import save
+ spec = catalog()
+ progress = progress or (lambda value: None)
+ check = check or (lambda: None)
+ if not (spec.get('modelscope') or spec.get('huggingface')):
+ raise PackError('not_published')
+ wheels = wheel_rows(spec)
+ if wheels is None:
+ raise PackError('unsupported')
+ folder = directory(root)
+ folder.mkdir(parents=True, exist_ok=True)
+ cache = wheels_folder(root)
+ cache.mkdir(parents=True, exist_ok=True)
+ ledger = _ledger(folder)
+ cached = _ledger(cache)
+ # A file without a stamp (a copied folder) is hashed once here and stamped.
+ todo = [row for row in rows(spec) if not provision.verified(folder / row['file'], row, ledger)]
+ save(folder / LEDGER, ledger)
+ fetch = [] if _unpacked(root, spec) else [row for row in wheels if not provision.verified(cache / row['file'], row, cached)]
+ save(cache / LEDGER, cached)
+ total = sum(row['bytes'] for row in todo) + sum(row['bytes'] for row in fetch)
+ retained = 0
+ for row, where in [(row, folder) for row in todo] + [(row, cache) for row in fetch]:
+ partial = (where / row['file']).with_suffix((Path(row['file']).suffix) + '.partial')
+ if partial.is_file() and partial.stat().st_size < row['bytes']:
+ retained += partial.stat().st_size
+ if total and disk_space.free_bytes(folder) < total - retained + RESERVE:
+ raise PackError('disk_space')
+ plan = _download_plan(root)
+ # The first dubbing download follows the same hosts as the image model; installs whose plan has
+ # no entry for the family try ModelScope first, then the two Hugging Face hosts.
+ sources = plan['sources'] = dict(plan.get('sources', {}))
+ if not sources.get(FAMILY):
+ sources[FAMILY] = [{'id': 'modelscope'}] + [
+ dict(item) for item in sources.get('models', []) if item['id'] in ('official', 'hf-mirror')]
+ if len(sources[FAMILY]) == 1:
+ sources[FAMILY] += [{'id': 'official'}, {'id': 'hf-mirror'}]
+ plan.update(quiet=True, resource_check=check)
+ wheel_plan = runtime_plan(plan)
+ done = 0
+
+ def transfer(row, path, remote_row, ledger_file, stamps, plan=plan):
+ nonlocal done
+ check()
+ path.parent.mkdir(parents=True, exist_ok=True)
+
+ def moved(current, size, speed, **kwargs):
+ progress(dict(phase='download', done=done + current, total=total))
+ network.download(network.model_urls(plan, remote_row), path, row['sha256'], moved,
+ network=plan, size=row['bytes'], category=FAMILY, keep_partial=True,
+ stall_seconds=30, slow_seconds=15, low_speed_limit=64 * 1024)
+ stamps[str(path)] = provision.file_identity(path, row)
+ save(ledger_file, stamps)
+ done += row['bytes']
+ try:
+ for row in fetch:
+ transfer(row, cache / row['file'], runtime_remote(row), cache / LEDGER, cached, plan=wheel_plan)
+ if fetch or not _unpacked(root, spec):
+ progress(dict(phase='unpack', done=done, total=total))
+ _unpack(root, wheels, check)
+ for row in todo:
+ transfer(row, folder / row['file'], remote(spec, row), folder / LEDGER, ledger)
+ except PackError:
+ raise
+ except InterruptedError:
+ raise
+ except ValueError as error:
+ raise PackError('disk_space' if str(error) == 'disk_space' else 'download_failed') from error
+ except OSError as error:
+ raise PackError('disk_space' if getattr(error, 'errno', None) == 28 else 'download_failed') from error
+ except Exception as error:
+ raise PackError('download_failed') from error
diff --git a/freevideo_engine/dubbing/reference.py b/freevideo_engine/dubbing/reference.py
new file mode 100644
index 0000000..89ea87a
--- /dev/null
+++ b/freevideo_engine/dubbing/reference.py
@@ -0,0 +1,39 @@
+"""The voice reference of a dubbing job: an audio file, or the sound of a video, as a short mono wav."""
+import numpy as np
+
+from . import backends, wavio
+
+RATE = 24000
+MIN_SECONDS = 1.0
+
+
+def extract(source, target, start=0., seconds=backends.MAX_REFERENCE_SECONDS):
+ """Decode up to `seconds` of the first audio stream of `source` (audio or video) from `start`
+ into a mono 16-bit wav at `target`. Returns the seconds written."""
+ import av
+ chunks = []
+ wanted = int(seconds * RATE)
+ got = 0
+ with av.open(str(source)) as container:
+ if not container.streams.audio:
+ raise ValueError('no_audio')
+ stream = container.streams.audio[0]
+ if start > 0:
+ container.seek(int(start / stream.time_base) if stream.time_base else 0, stream=stream)
+ resampler = av.AudioResampler(format='flt', layout='mono', rate=RATE)
+ for frame in container.decode(stream):
+ if frame.time is not None and frame.time + frame.samples / max(frame.rate, 1) <= start:
+ continue
+ for part in resampler.resample(frame):
+ chunk = part.to_ndarray().reshape(-1)
+ chunks.append(chunk)
+ got += len(chunk)
+ if got >= wanted:
+ break
+ if not chunks:
+ raise ValueError('no_audio')
+ samples = np.concatenate(chunks)[:wanted]
+ if len(samples) < MIN_SECONDS * RATE:
+ raise ValueError('reference_short')
+ wavio.write(target, samples, RATE)
+ return len(samples) / RATE
diff --git a/freevideo_engine/dubbing/service.py b/freevideo_engine/dubbing/service.py
new file mode 100644
index 0000000..b4fc861
--- /dev/null
+++ b/freevideo_engine/dubbing/service.py
@@ -0,0 +1,508 @@
+"""Dubbing for the editor page, the Agent and the voice change: one job at a time, one interface.
+
+A job takes lines (text and target seconds) and one reference voice, and writes one wav per line
+into ComfyUI's input folder (`freevideo-dubbing/<job>/`), where the other tools already find media.
+The model runs in its own process, so its memory leaves with it. The model is downloaded on the
+first use after the user agreed (`install`).
+"""
+import json
+import logging
+import os
+from pathlib import Path
+import secrets
+import shutil
+import subprocess
+import threading
+import time
+import uuid
+
+from . import backends, pack
+from ..prompt_vlm.service import Worker
+
+OUTPUT = 'freevideo-dubbing'
+MAX_LINES = 500
+MAX_TEXT = 2000
+MAX_SECONDS = 120.
+MIN_SECONDS = .3
+RUN_SECONDS = 300
+LINE_SECONDS = 45
+RUN_LIMIT = 3600
+KEPT_RUNS = 8
+LINE_KEYS = ('id', 'seconds', 'target', 'samples', 'onset', 'offset', 'onset_error', 'offset_error', 'error', 'speed',
+ 'passes', 'stretch', 'fitted', 'warning', 'overrun', 'gain_db', 'reference', 'code')
+CODES = {'invalid_request', 'unsupported_media', 'busy', 'model_missing', 'consent_required', 'unknown_job',
+ 'reference_text_required', 'reference_short', 'no_audio'}
+KNOWN = {'disk_space', 'download_failed', 'not_published', 'unsupported', 'timeout', 'worker_exit', 'worker_cleanup_failed',
+ 'out_of_memory', 'model_missing', 'invalid_request', 'reference_text_required', 'reference_short'}
+
+
+def reference_of(value, *, required):
+ """{file, text, start[, end]} of a job or line reference; None when absent and not required."""
+ if value is None and not required:
+ return None
+ if not isinstance(value, dict) or not isinstance(value.get('file'), str) or not value['file']:
+ raise ValueError('invalid_request')
+ text = value.get('text', '')
+ start = value.get('start', 0)
+ end = value.get('end')
+ if (not isinstance(text, str) or len(text) > 1000 or isinstance(start, bool)
+ or not isinstance(start, (int, float)) or not 0 <= start <= 36000):
+ raise ValueError('invalid_request')
+ result = dict(file=value['file'], text=text.strip(), start=float(start))
+ if end is not None:
+ if isinstance(end, bool) or not isinstance(end, (int, float)) or not start < end <= 36000:
+ raise ValueError('invalid_request')
+ result['end'] = float(end)
+ return result
+
+
+def request(value):
+ """The job a page, the Agent or the voice change may send; anything else is refused."""
+ if not isinstance(value, dict):
+ raise ValueError('invalid_request')
+ lines = value.get('lines')
+ if not isinstance(lines, list) or not 0 < len(lines) <= MAX_LINES:
+ raise ValueError('invalid_request')
+ clean = []
+ seen = set()
+ for index, line in enumerate(lines):
+ if not isinstance(line, dict):
+ raise ValueError('invalid_request')
+ text = line.get('text')
+ seconds = line.get('seconds')
+ name = str(line.get('id', 'line%03d' % index))
+ if (not isinstance(text, str) or not text.strip() or len(text) > MAX_TEXT
+ or isinstance(seconds, bool) or not isinstance(seconds, (int, float))
+ or not MIN_SECONDS <= seconds <= MAX_SECONDS
+ or not name or len(name) > 64 or not all(c.isalnum() or c in '-_' for c in name)
+ or name in seen):
+ raise ValueError('invalid_request')
+ seen.add(name)
+ row = dict(id=name, text=text.strip(), seconds=float(seconds))
+ onset, offset = line.get('onset'), line.get('offset')
+ for key, number in (('onset', onset), ('offset', offset)):
+ if number is None:
+ continue
+ if isinstance(number, bool) or not isinstance(number, (int, float)) or not 0 <= number <= seconds + 1e-9:
+ raise ValueError('invalid_request')
+ row[key] = float(number)
+ if row.get('offset', seconds) - row.get('onset', 0.) < MIN_SECONDS / 2:
+ raise ValueError('invalid_request')
+ if line.get('reference') is not None:
+ row['reference'] = reference_of(line['reference'], required=True)
+ instruction = line.get('instruction')
+ if instruction not in (None, ''):
+ if not isinstance(instruction, str) or len(instruction) > 200:
+ raise ValueError('invalid_request')
+ row['instruction'] = instruction.strip()
+ clean.append(row)
+ reference = reference_of(value.get('reference'), required=not all('reference' in row for row in clean))
+ result = dict(lines=clean)
+ if reference is not None:
+ result['reference'] = reference
+ seed = value.get('seed')
+ if seed is not None:
+ if isinstance(seed, bool) or not isinstance(seed, int) or not 0 <= seed < 2**31:
+ raise ValueError('invalid_request')
+ result['seed'] = seed
+ name = value.get('backend')
+ if name is not None:
+ if name not in backends.names():
+ raise ValueError('invalid_request')
+ result['backend'] = name
+ return result
+
+
+class Service:
+ def __init__(self, installation, input_directory):
+ self.installation = installation
+ self.input_directory = input_directory
+ self.mutex = threading.Lock()
+ self.stop = threading.Event()
+ self.state = dict(phase='idle')
+ self.key = None
+ self.thread = None
+ self.process = None
+ self.result_expires = 0.
+ self.report = {}
+ self.closed = False
+ import atexit
+ atexit.register(self.close)
+
+ def info(self, wait=True):
+ root, _ = self.installation()
+ try:
+ state = pack.status(root)
+ except (OSError, ValueError):
+ return dict(ready=False, bytes=0, busy=self.busy(), backends=backends.names())
+ if wait and not state['ready']:
+ state['ready'] = pack.settle(root)
+ if state['ready']:
+ state['bytes'] = 0
+ return dict(ready=state['ready'], model=pack.NAME, license=state['license'], bytes=state['bytes'],
+ published=state['published'], supported=state['supported'], busy=self.busy(), backends=backends.names())
+
+ def busy(self):
+ return self.thread is not None and self.thread.is_alive()
+
+ def check(self):
+ if self.stop.is_set():
+ raise InterruptedError('cancelled')
+
+ def progress(self, row):
+ with self.mutex:
+ self.state.update(row)
+
+ def start(self, action, value):
+ if action not in ('install', 'dub'):
+ raise ValueError('invalid_request')
+ if self.busy():
+ raise ValueError('busy')
+ root, machine = self.installation()
+ if action == 'install':
+ if not isinstance(value, dict) or value.get('accept_download') is not True:
+ raise ValueError('consent_required')
+ else:
+ if not pack.settle(root):
+ raise ValueError('model_missing')
+ value = request(value)
+ for item in [value.get('reference')] + [line.get('reference') for line in value['lines']]:
+ if item:
+ item['path'] = self.resolve(item['file'])
+ with self.mutex:
+ if self.busy():
+ raise ValueError('busy')
+ self.key = secrets.token_urlsafe(24)
+ self.stop.clear()
+ self.result_expires = 0.
+ self.state = dict(phase='starting')
+ self.report = {}
+ self.thread = threading.Thread(target=self.work, args=(root, machine, action, value),
+ name='freevideo-dubbing', daemon=True)
+ self.thread.start()
+ return dict(job=self.key)
+
+ def resolve(self, name):
+ """A reference inside ComfyUI's input folder; audio, or a video whose sound is used."""
+ from ..comfy_assets import media_kind, resolve_assets
+ resolved = resolve_assets(json.dumps([dict(file=name, role='reference')]), self.input_directory())
+ path = resolved['references'][0]['path']
+ if media_kind(path) not in ('audio', 'video'):
+ raise ValueError('unsupported_media')
+ return path
+
+ def status(self, key):
+ if not isinstance(key, str) or not self.key or not secrets.compare_digest(key, self.key):
+ raise ValueError('unknown_job')
+ with self.mutex:
+ if self.result_expires and time.monotonic() > self.result_expires:
+ self.state.pop('lines', None)
+ self.state.update(phase='failed', error='expired')
+ row = dict(self.state)
+ # Do not announce completion while the runtime lease is still held.
+ if self.busy() and row.get('phase') in ('complete', 'failed', 'cancelled'):
+ row = dict(phase='finishing')
+ return row
+
+ def cancel(self, key):
+ self.status(key)
+ self.stop.set()
+ return dict(phase='cancelling')
+
+ def close(self):
+ self.closed = True
+ self.stop.set()
+ process = self.process
+ if process is not None and process.poll() is None:
+ from .. import processes
+ processes.stop(process, grace=5)
+
+ @staticmethod
+ def release_others():
+ """Give the GPU back: compile, idle image worker, resident engine process, ComfyUI models."""
+ from ..background_compile import COMPILE
+ COMPILE.stop()
+ try:
+ from server import PromptServer
+ from ..image_gen.service import release_idle_worker
+ release_idle_worker(PromptServer.instance)
+ except Exception:
+ pass
+ from ..resident_process import OWNER
+ if OWNER.active:
+ raise BlockingIOError('busy')
+ OWNER.stop_prewarm()
+ from ..encoder_prewarm import IDLE
+ IDLE.stop()
+ from ..comfy_residency import release_native
+ from ..image_gen.service import comfy_server
+ try:
+ release_native(comfy_server())
+ except Exception: # only frees memory early; the worker's own allocation still decides
+ logging.warning('[FreeVideo] Could not unload ComfyUI models before dubbing', exc_info=True)
+
+ def output_folder(self, key_id):
+ base = Path(self.input_directory()) / OUTPUT
+ base.mkdir(parents=True, exist_ok=True)
+ # Keep the latest few runs; older ones are the page's old results.
+ runs = sorted((p for p in base.iterdir() if p.is_dir()), key=lambda p: p.stat().st_mtime)
+ for old in runs[:max(0, len(runs) - KEPT_RUNS + 1)]:
+ shutil.rmtree(old, ignore_errors=True)
+ folder = base / key_id
+ folder.mkdir()
+ return folder
+
+ def work(self, root, machine, action, value):
+ from ..locking import runtime_lock
+ from ..monitoring import save
+ started = time.monotonic()
+ try:
+ if action == 'install':
+ with runtime_lock(Path(root) / 'dubbing-install.lock', inherit=False):
+ self.check()
+ pack.install(root, self.progress, self.check)
+ self.check()
+ self.progress(dict(phase='complete'))
+ else:
+ self.release_others()
+ with runtime_lock(Path(root) / 'engine.lock', inherit=False) as descriptor:
+ self.check()
+ self.dub(root, machine, descriptor, value)
+ except InterruptedError:
+ self.progress(dict(phase='cancelled'))
+ except BlockingIOError:
+ self.progress(dict(phase='failed', error='busy'))
+ except pack.PackError as error:
+ self.progress(dict(phase='failed', error=error.reason))
+ except Exception as error:
+ reason = str(error)
+ self.progress(dict(phase='failed', exception=type(error).__name__,
+ error=reason if reason in KNOWN else 'download_failed' if action == 'install'
+ else 'dub_failed'))
+ finally:
+ if self.stop.is_set():
+ self.progress(dict(phase='cancelled'))
+ self.result_expires = time.monotonic() + 600
+ # Allowlisted metadata only: never the text, file names or paths.
+ report = {key: item for key, item in self.state.items()
+ if key in ('phase', 'error', 'exception', 'timings', 'worker_returncode', 'backend',
+ 'sample_rate', 'reference_seconds', 'loaded_seconds')}
+ if self.state.get('lines'):
+ report['lines'] = [{key: row.get(key) for key in LINE_KEYS if key not in ('id', 'reference')}
+ for row in self.state['lines']]
+ report.update(schema='freevideo.dubbing', schema_version=1, action=action, created_epoch=time.time(),
+ report_id=uuid.uuid4().hex, elapsed_seconds=time.monotonic() - started)
+ self.report = report
+ try:
+ save(Path(root) / 'dubbing-report.json', report)
+ except OSError:
+ pass
+
+ @staticmethod
+ def environment(root):
+ from ..comfy_environment import isolated_environment
+ from ..comfy_bridge import source_root
+ from .. import processes, resident_process
+ env = isolated_environment(root, source_root())
+ for name in ('FREEVIDEO_RUNTIME_LOCK_FD', processes.HANDLE_ENV, resident_process.ENV):
+ env.pop(name, None)
+ return env
+
+ @staticmethod
+ def spawn(root, machine, env, descriptor=None):
+ from .. import processes
+ command = processes.module_command('freevideo_engine.dubbing.worker', pack.directory(root), pack.runtime_path(root) or '')
+ command[0] = machine['comfy_python']
+ return processes.popen(command, env=env, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE, text=True, encoding='utf-8', errors='replace',
+ pass_fds=() if descriptor is None else (descriptor,),
+ supervise=True, start_new_session=True)
+
+ def dub(self, root, machine, descriptor, value):
+ from .. import processes, resident_process
+ from . import reference
+ import psutil
+ env = self.environment(root)
+ key_id = uuid.uuid4().hex[:12]
+ folder = self.output_folder(key_id)
+ self.progress(dict(phase='preparing'))
+ extracted = {}
+
+ def extract(ref, name):
+ key = (ref['path'], ref['start'], ref.get('end'))
+ if key not in extracted:
+ wav = folder / name
+ span = ref['end'] - ref['start'] if ref.get('end') else backends.MAX_REFERENCE_SECONDS
+ try:
+ reference.extract(ref['path'], wav, start=ref['start'], seconds=min(span, backends.MAX_REFERENCE_SECONDS))
+ except ValueError as error:
+ if str(error) not in CODES:
+ raise ValueError('unsupported_media') from None
+ extracted[key] = str(error)
+ return extracted[key]
+ except Exception:
+ raise ValueError('unsupported_media') from None
+ extracted[key] = str(wav)
+ return extracted[key]
+ job = dict(value, output_dir=str(folder))
+ if value.get('reference'):
+ made = extract(value['reference'], 'reference.wav')
+ if made not in (str(folder / 'reference.wav'),):
+ raise ValueError(made)
+ job['reference'] = dict(path=made, text=value['reference']['text'])
+ job['lines'] = []
+ for index, line in enumerate(value['lines']):
+ row = {key: item for key, item in line.items() if key != 'reference'}
+ if line.get('reference'):
+ made = extract(line['reference'], 'reference-%d.wav' % index)
+ if os.path.isfile(made):
+ row['reference'] = dict(path=made, text=line['reference']['text'])
+ elif made == 'reference_short' and job.get('reference'):
+ # the job's reference is used, and the line says so
+ row['reference_short'] = True
+ else:
+ raise ValueError(made)
+ job['lines'].append(row)
+ session = Worker(self.spawn(root, machine, env, descriptor))
+ child = session.process
+ self.process = child
+ result = {}
+ lines = []
+ write_errors = []
+
+ def submit():
+ try:
+ child.stdin.write(json.dumps(job, ensure_ascii=True))
+ child.stdin.close()
+ except Exception as error:
+ write_errors.append(error)
+ writer = threading.Thread(target=submit, daemon=True)
+
+ def read_rows():
+ import queue
+ while True:
+ try:
+ row = session.rows.get_nowait()
+ except queue.Empty:
+ return
+ phase = row.get('phase')
+ if phase in ('complete', 'failed', 'cancelled'):
+ result.update(row)
+ elif phase == 'line':
+ entry = {key: row[key] for key in LINE_KEYS if key in row}
+ if row.get('file'):
+ entry['file'] = '%s/%s/%s' % (OUTPUT, key_id, row['file'])
+ lines.append(entry)
+ self.progress(dict(phase='synthesizing', index=row.get('index'), count=len(job['lines']),
+ lines=list(lines)))
+ elif phase in ('loading', 'loaded', 'synthesizing'):
+ self.progress(dict((k, v) for k, v in row.items() if k in ('phase', 'index', 'count', 'seconds')))
+ observed = {}
+ try:
+ writer.start()
+ deadline = time.monotonic() + min(RUN_LIMIT, RUN_SECONDS + LINE_SECONDS * len(job['lines']))
+ while child.poll() is None:
+ read_rows()
+ self.check()
+ if write_errors:
+ raise write_errors[0]
+ if time.monotonic() > deadline:
+ raise TimeoutError('timeout')
+ try:
+ for process in psutil.Process(child.pid).children(recursive=True):
+ observed[process.pid] = process
+ except psutil.Error:
+ pass
+ self.stop.wait(.25)
+ session.reader.join(timeout=5)
+ read_rows()
+ self.check()
+ if session.reader.is_alive() or not result:
+ raise RuntimeError('worker_exit')
+ if result.get('phase') == 'failed':
+ raise RuntimeError(result.get('code') if result.get('code') in KNOWN else 'dub_failed')
+ if result.get('phase') == 'cancelled':
+ raise InterruptedError('cancelled')
+ if child.returncode:
+ raise RuntimeError('worker_exit')
+ self.progress(dict(phase='complete', lines=lines, backend=result.get('backend'),
+ sample_rate=result.get('sample_rate'),
+ reference_seconds=result.get('reference_seconds'),
+ worker_returncode=child.returncode))
+ finally:
+ if child.poll() is None:
+ processes.stop(child, grace=15)
+ child.wait()
+ writer.join(timeout=5)
+ session.close()
+ self.process = None
+ for wav in folder.glob('reference*.wav'):
+ try:
+ wav.unlink()
+ except OSError:
+ pass
+ surviving = []
+ for process in observed.values():
+ try:
+ if process.is_running() and process.status() != 'zombie':
+ surviving.append(process)
+ except psutil.NoSuchProcess:
+ pass
+ for process in surviving:
+ try:
+ process.kill()
+ except psutil.NoSuchProcess:
+ pass
+ if surviving:
+ psutil.wait_procs(surviving, timeout=5)
+ self.progress(dict(error='worker_cleanup_failed'))
+ # Failed or cancelled runs leave nothing behind.
+ if self.state.get('phase') != 'complete':
+ shutil.rmtree(folder, ignore_errors=True)
+
+
+def register():
+ import asyncio
+ from aiohttp import web
+ from server import PromptServer
+ import folder_paths
+ from ..comfy_bridge import installation
+ server = PromptServer.instance
+ if server is None or getattr(server, '_freevideo_dubbing', None) is not None:
+ return
+ service = Service(installation, folder_paths.get_input_directory)
+ server._freevideo_dubbing = service
+ token = secrets.token_urlsafe(24)
+
+ @server.routes.get('/freevideo/dubbing')
+ async def info(request):
+ try:
+ return web.json_response(dict(await asyncio.to_thread(service.info, False), token=token))
+ except (OSError, ValueError):
+ return web.json_response(dict(error='setup_required'), status=400)
+
+ @server.routes.post('/freevideo/dubbing/{action}')
+ async def action(request):
+ if not secrets.compare_digest(request.headers.get('X-FreeVideo-Dubbing', ''), token):
+ return web.json_response(dict(error='reload'), status=403)
+ try:
+ name = request.match_info['action']
+ if request.content_length and request.content_length > 512 * 1024:
+ raise ValueError('invalid_request')
+ value = await request.json()
+ if not isinstance(value, dict):
+ raise ValueError('invalid_request')
+ if name == 'status':
+ result = service.status(value.get('job'))
+ elif name == 'cancel':
+ result = service.cancel(value.get('job'))
+ else:
+ running, pending = server.prompt_queue.get_current_queue()
+ if running or pending:
+ raise ValueError('busy')
+ if name == 'dub':
+ await asyncio.to_thread(pack.settle, service.installation()[0])
+ result = service.start(name, value)
+ return web.json_response(result)
+ except (ValueError, OSError) as error:
+ return web.json_response(dict(error=str(error) if str(error) in CODES else 'invalid_media'), status=400)
diff --git a/freevideo_engine/dubbing/wavio.py b/freevideo_engine/dubbing/wavio.py
new file mode 100644
index 0000000..9112b24
--- /dev/null
+++ b/freevideo_engine/dubbing/wavio.py
@@ -0,0 +1,27 @@
+"""16-bit PCM wav files, read and written without an audio library."""
+import wave
+
+import numpy as np
+
+
+def read(path):
+ """Float32 samples shaped (frames, channels) and the sample rate of a 16-bit PCM wav."""
+ with wave.open(str(path), 'rb') as file:
+ if file.getsampwidth() != 2:
+ raise ValueError('unsupported_wav')
+ channels, rate = file.getnchannels(), file.getframerate()
+ raw = file.readframes(file.getnframes())
+ data = np.frombuffer(raw, dtype='<i2').astype(np.float32) / 32768.
+ return data.reshape(-1, channels), rate
+
+
+def write(path, samples, rate):
+ """Write mono (n,) or (n, channels) float samples in [-1, 1] as 16-bit PCM."""
+ data = np.asarray(samples, dtype=np.float32)
+ channels = 1 if data.ndim == 1 else data.shape[1]
+ pcm = np.round(np.clip(data, -1., 1.) * 32767.).astype('<i2')
+ with wave.open(str(path), 'wb') as file:
+ file.setnchannels(channels)
+ file.setsampwidth(2)
+ file.setframerate(int(rate))
+ file.writeframes(pcm.tobytes())
diff --git a/freevideo_engine/dubbing/worker.py b/freevideo_engine/dubbing/worker.py
new file mode 100644
index 0000000..8bca359
--- /dev/null
+++ b/freevideo_engine/dubbing/worker.py
@@ -0,0 +1,297 @@
+"""One dubbing job per process: the model, its CUDA context and its memory leave with the process.
+
+Arguments: the model folder, then the folder of the downloaded runtime libraries (optional).
+The request is one JSON object on stdin; progress and the result are JSON lines on stdout. A line is
+fitted to its target duration by `align.fit`, then written as a wav file in `output_dir`.
+"""
+import json
+import math
+import os
+import sys
+import time
+import traceback
+
+from . import align, backends
+
+IDLE_SECONDS = 600
+ROW_LIMIT = 128 * 1024
+
+
+def emit(**row):
+ sys.stdout.write(json.dumps(row, ensure_ascii=False, separators=(',', ':')) + '\n')
+ sys.stdout.flush()
+
+
+LEVEL_TARGET = 0.06
+LEVEL_PEAK = 0.95
+LEVEL_GAIN = 10.
+
+
+def leveled(data, rate=None):
+ """The model gives back the level of its reference, so a quiet recording would come back nearly
+ silent. Samples quieter than LEVEL_TARGET (RMS) are raised towards it: the gain is 1 at the target and
+ grows smoothly as the recording gets quieter, never below 1, never past LEVEL_GAIN, never into clipping.
+ With `rate` the level is that of the voiced part, so the silence around a line cut from a separated
+ voice does not raise the line more than its voice needs; a recording that is loud as a whole is left
+ as it is."""
+ import numpy as np
+ rms = float(np.sqrt(np.mean(np.square(data, dtype=np.float64)))) if len(data) else 0.
+ peak = float(np.max(np.abs(data))) if len(data) else 0.
+ if rms <= 1e-5 or rms >= LEVEL_TARGET:
+ return data, 1.
+ if rate:
+ rms = voiced_level(data, rate) or rms
+ gain = max(1., min(LEVEL_TARGET / rms, LEVEL_PEAK / peak, LEVEL_GAIN))
+ if gain == 1.:
+ return data, 1.
+ return data * np.float32(gain), gain
+
+
+MATCH_RANGE = (0.1, 10.)
+
+
+def voiced_level(data, rate):
+ """RMS of the voiced part of `data` (None when there is none)."""
+ import numpy as np
+ span = align.voiced_span(data, rate)
+ if span is None:
+ return None
+ part = np.asarray(data[span[0]:span[1]], dtype=np.float64)
+ return float(np.sqrt(np.mean(np.square(part)))) if len(part) else None
+
+
+def matched(samples, rate, onset, offset, level):
+ """`samples` scaled so that the voiced part (from `onset` to `offset` seconds) has RMS `level`: the line
+ comes back as loud as the original line it replaces, whatever level the model was given. Returns
+ (samples, gain); the gain stays inside MATCH_RANGE and never clips."""
+ import numpy as np
+ current = voiced_level(samples[int(onset * rate):int(offset * rate)], rate)
+ peak = float(np.max(np.abs(samples))) if len(samples) else 0.
+ if not level or not current or peak <= 0:
+ return samples, 1.
+ gain = min(max(level / current, MATCH_RANGE[0]), MATCH_RANGE[1], LEVEL_PEAK / peak)
+ if abs(gain - 1.) < 0.01:
+ return samples, 1.
+ return samples * np.float32(gain), gain
+
+
+CLAUSE_END = '。!?;,、.!?;,\n'
+
+
+def cut_text(text, share):
+ """The longest start of `text` that is no more than `share` of it and ends where a clause or a word
+ ends; returns (start of the text, its share of the whole text)."""
+ limit = int(len(text) * share)
+ if limit <= 0:
+ return '', 0.
+ stops = [i for i in range(limit, 0, -1) if text[i - 1] in CLAUSE_END or text[i - 1] == ' ' or text[i:i + 1] == ' ']
+ end = next((i for i in stops if i >= limit // 2), limit)
+ kept = text[:end].strip()
+ return kept, len(kept) / len(text)
+
+
+def usable_reference(path, folder, name='reference-prepared.wav', text='', texts=None):
+ """Return a mono wav no longer than the models accept and not too quiet (a changed copy if either
+ applies), and its seconds. A reference that is too long is cut where its text is cut (the model reads the
+ text it is given, so a text longer than the audio is spoken into the line): the cut text goes to
+ `texts[name]`."""
+ from . import wavio
+ import numpy as np
+ data, rate = wavio.read(path)
+ seconds = len(data) / rate
+ if seconds < 1.0:
+ raise ValueError('reference_short')
+ changed = False
+ if texts is not None:
+ texts[name] = text
+ if seconds > backends.MAX_REFERENCE_SECONDS:
+ end = int(backends.MAX_REFERENCE_SECONDS * rate)
+ if text:
+ kept, share = cut_text(text, backends.MAX_REFERENCE_SECONDS / seconds)
+ if kept:
+ text, end = kept, int(len(data) * share)
+ # the cut falls on the quietest 10 ms of the last half second
+ frame = max(1, rate // 100)
+ first = max(0, end - rate // 2) // frame
+ count = (end - first * frame) // frame
+ if count > 1:
+ rms = np.sqrt(np.square(np.asarray(data[first * frame:(first + count) * frame], dtype=np.float64))
+ .reshape(count, frame).mean(axis=1))
+ end = (first + int(np.argmin(rms))) * frame + frame // 2
+ if texts is not None:
+ texts[name] = text
+ data = data[:end]
+ seconds = len(data) / rate
+ changed = True
+ data, gain = leveled(data, rate)
+ if gain != 1.:
+ changed = True
+ if not changed:
+ return path, seconds
+ target = os.path.join(folder, name)
+ wavio.write(target, data, rate)
+ return target, seconds
+
+
+def validate(request):
+ lines = request.get('lines')
+ if not isinstance(lines, list) or not lines or len(lines) > 500:
+ raise ValueError('invalid_request')
+ for line in lines:
+ if not isinstance(line, dict) or not str(line.get('text') or '').strip():
+ raise ValueError('invalid_request')
+ if not float(line.get('seconds') or 0) > 0:
+ raise ValueError('invalid_request')
+ reference = request.get('reference') or {}
+ own = [(line.get('reference') or {}) for line in lines]
+ if any(not isinstance(item, dict) for item in own):
+ raise ValueError('invalid_request')
+ for item in own:
+ if item and not os.path.isfile(str(item.get('path') or '')):
+ raise ValueError('invalid_request')
+ if (reference or not all(own)) and not os.path.isfile(str(reference.get('path') or '')):
+ raise ValueError('invalid_request')
+ if not os.path.isdir(str(request.get('output_dir') or '')):
+ raise ValueError('invalid_request')
+ return lines, reference
+
+
+def run(request, model_dir, device='cuda'):
+ from . import wavio
+ from ..prompt_vlm.worker import Cancelled
+ lines, reference = validate(request)
+ folder = request['output_dir']
+ name = request.get('backend') or backends.DEFAULT
+ seed = request.get('seed')
+ # References are prepared before the model loads: a bad one is refused in seconds, not after the load.
+ prepared = {}
+ levels = {}
+ cut = {}
+
+ def prepare(source, key):
+ if key not in prepared:
+ name = 'reference-%s.wav' % key
+ prepared[key] = usable_reference(source['path'], folder, name, str(source.get('text') or '').strip(), cut)
+ data, rate = wavio.read(source['path'])
+ levels[key] = voiced_level(data, rate)
+ return prepared[key]
+ job_ref = None
+ if reference:
+ job_ref = prepare(reference, 'job') + (cut['reference-job.wav'],)
+ used = []
+ match = {} # per line with its own reference: the voiced RMS the line is to come back with
+ for index, line in enumerate(lines):
+ own = line.get('reference')
+ if own:
+ try:
+ used.append(('line',) + prepare(own, 'line%d' % index) + (cut['reference-line%d.wav' % index], None))
+ match[index] = levels.get('line%d' % index)
+ continue
+ except ValueError as error:
+ if str(error) != 'reference_short' or job_ref is None:
+ raise
+ used.append(('job',) + job_ref + ('reference_short',))
+ continue
+ used.append(('job',) + job_ref + ('reference_short' if line.get('reference_short') else None,))
+ emit(phase='loading')
+ started = time.monotonic()
+ backend = backends.create(name, model_dir, device=device)
+ emit(phase='loaded', seconds=round(time.monotonic() - started, 2))
+ try:
+ if backend.needs_reference_text and any(not item[3] for item in used):
+ raise ValueError('reference_text_required')
+ results = []
+ for index, line in enumerate(lines):
+ emit(phase='synthesizing', index=index, count=len(lines))
+ source, ref_path, ref_seconds, ref_text, note = used[index]
+ if seed is not None:
+ from .cosyvoice3.utils.common import set_all_random_seed
+ set_all_random_seed(int(seed) + index)
+ instruction = (line.get('instruction') or '').strip() or None
+ text = str(line['text']).strip()
+ target = float(line['seconds'])
+ row = dict(id=line.get('id'), target=target, reference=source)
+ try:
+ def synth(speed, text=text, instruction=instruction, ref_path=ref_path, ref_text=ref_text):
+ return backend.synth(text, ref_path, ref_text, speed, instruction)
+
+ fitted = align.fit(synth, target, onset=float(line.get('onset') or 0.),
+ offset=None if line.get('offset') is None else float(line['offset']))
+ except Cancelled:
+ raise
+ except Exception as error:
+ traceback.print_exc(file=sys.stderr)
+ row.update(failure(error))
+ results.append(row)
+ emit(phase='line', index=index, **row)
+ continue
+ samples, gain = fitted['samples'], 1.
+ if match.get(index):
+ samples, gain = matched(samples, fitted['rate'], fitted['onset'], fitted['offset'], match[index])
+ path = os.path.join(folder, '%s.wav' % (line.get('id') or 'line%03d' % index))
+ wavio.write(path, samples, fitted['rate'])
+ row.update(file=os.path.basename(path), seconds=round(fitted['seconds'], 4), samples=len(fitted['samples']),
+ onset=round(fitted['onset'], 3), offset=round(fitted['offset'], 3),
+ onset_error=round(fitted['onset_error'], 3), offset_error=round(fitted['offset_error'], 3),
+ error=round(fitted['error'], 3), speed=round(fitted['speed'], 3), passes=fitted['passes'],
+ stretch=round(fitted['stretch'], 3), fitted=bool(fitted['fitted']),
+ warning=fitted['warning'] or (note if note == 'reference_short' else None),
+ overrun=round(fitted['overrun'], 3), gain_db=round(20 * math.log10(gain), 1))
+ results.append(row)
+ emit(phase='line', index=index, **row)
+ finally:
+ backend.close()
+ if all('code' in row for row in results):
+ raise _LineFailure(results[0])
+ return dict(lines=results, reference_seconds=round(used[0][2], 2), backend=name, sample_rate=backend.sample_rate)
+
+
+class _LineFailure(Exception):
+ """Every line failed: the job fails with the first line's code."""
+ def __init__(self, row):
+ super().__init__(row['code'])
+ self.row = row
+
+
+CODES = ('invalid_request', 'reference_text_required', 'reference_short')
+
+
+def failure(error):
+ if isinstance(error, _LineFailure):
+ return dict(code=error.row['code'], **({'error': error.row['error']} if 'error' in error.row else {}))
+ text = str(error)
+ if isinstance(error, ValueError) and text in CODES:
+ return dict(code=text)
+ if isinstance(error, FileNotFoundError):
+ return dict(code='model_missing')
+ low = text.lower()
+ if isinstance(error, MemoryError) or 'out of memory' in low:
+ return dict(code='out_of_memory')
+ return dict(code='failed', error=type(error).__name__)
+
+
+def main():
+ from ..prompt_vlm.worker import Cancelled, stop_on_signal
+ stop_on_signal()
+ try:
+ request = json.loads(sys.stdin.buffer.read(1024 * 1024))
+ if not isinstance(request, dict):
+ raise ValueError('invalid_request')
+ if len(sys.argv) > 2 and sys.argv[2]:
+ # -I ignores PYTHONPATH: the downloaded runtime (onnxruntime) is handed over as an argument.
+ from .pack import activate
+ activate(sys.argv[2])
+ result = run(request, sys.argv[1], device=request.get('device') or 'cuda')
+ emit(phase='complete', **result)
+ return 0
+ except Cancelled:
+ emit(phase='cancelled')
+ return 0
+ except Exception as error:
+ traceback.print_exc(file=sys.stderr)
+ emit(phase='failed', **failure(error))
+ return 1
+
+
+if __name__ == '__main__':
+ raise SystemExit(main())
diff --git a/freevideo_engine/licenses/CosyVoice-Apache-2.0.txt b/freevideo_engine/licenses/CosyVoice-Apache-2.0.txt
new file mode 100644
index 0000000..31e1c88
--- /dev/null
+++ b/freevideo_engine/licenses/CosyVoice-Apache-2.0.txt
@@ -0,0 +1,206 @@
+Copyright (c) 2024 Alibaba Inc (FunAudioLLM). CosyVoice, Fun-CosyVoice3-0.5B-2512 and the code in
+freevideo_engine/dubbing/cosyvoice3 adapted from https://github.com/FunAudioLLM/CosyVoice are licensed
+under the Apache License 2.0 reproduced below. FreeVideo reduced the code to the zero-shot and instruct2
+inputs and changed its file, audio and optional-dependency handling; each source file keeps its header.
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright 2026 FreeToken Authors
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/freevideo_engine/licenses/CosyVoice-NOTICE.txt b/freevideo_engine/licenses/CosyVoice-NOTICE.txt
new file mode 100644
index 0000000..482132a
--- /dev/null
+++ b/freevideo_engine/licenses/CosyVoice-NOTICE.txt
@@ -0,0 +1,29 @@
+Notices for code adapted into freevideo_engine/dubbing/cosyvoice3 (dubbing, optional download of the model).
+
+CosyVoice / Fun-CosyVoice3-0.5B-2512, Copyright (c) 2024 Alibaba Inc (FunAudioLLM): Apache License 2.0,
+see CosyVoice-Apache-2.0.txt. The model weights are downloaded on request and are not part of the installer.
+
+The following parts keep the MIT licence of their authors:
+- utils/mel.py mel_spectrogram, from Matcha-TTS (matcha/utils/audio.py), Copyright (c) 2023 Shivam Mehta,
+ derived from HiFi-GAN, Copyright (c) 2020 Jungil Kong.
+- flow/flow_matching.py attributes inlined from Matcha-TTS, Copyright (c) 2023 Shivam Mehta.
+- flow/DiT/modules.py rotary embedding, re-implemented from x_transformers and rotary-embedding-torch,
+ Copyright (c) 2020 Phil Wang.
+- transformer/activation.py Snake activation, from https://github.com/EdwardDixon/snake.
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
+associated documentation files (the "Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the
+following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial
+portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT
+LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO
+EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+USE OR OTHER DEALINGS IN THE SOFTWARE.
diff --git a/pyproject.toml b/pyproject.toml
index 448e585..51ea79d 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -35,5 +35,6 @@ include = ["freevideo_engine*"]
[tool.setuptools.package-data]
"freevideo_engine.prompt_vlm" = ["model.json"]
+"freevideo_engine.dubbing" = ["cosyvoice3_pack.json"]
"freevideo_engine.image_gen" = ["image_models.json", "licenses/NOTICE.txt"]
freevideo_engine = ["stage_estimate_params.json", "build-version.txt", "build-identity.json", "release_notes.json", "dependencies.json", "model_files.json", "model_shares.json", "prepared_models.json", "macos_reference_catalog.json", "superseded_tables.json", "bootstrap_versions.json", "curl_windows.json", "git_linux.json", "git_macos.json", "gcc_linux.json", "uv_downloads.json", "torch_downloads.json", "test_prompts.json", "windows_bootstrap.ps1", "assets/*.png", "assets/*.ico", "launcher/*.qml", "launcher/licenses/*.txt", "licenses/*.txt", "licenses/runtime/*.txt", "subtitle_pack.json", "frame_preview_weights.json", "agent/skill/freevideo/*.txt", "agent/skill/freevideo/references/*.txt", "depth_reference_model.json", "swap_lora.json"]