Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.
Add the dubbing engine: CosyVoice3 pack, worker, exact-length line placement, per-line reference...
Bowen Xuecd8f68d3d6d3
diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.mdindex 64906f6..647e47b 100644--- a/THIRD_PARTY_NOTICES.md+++ b/THIRD_PARTY_NOTICES.md@@ -17,6 +17,7 @@ FreeVideo code uses [Apache-2.0](LICENSE). Its dependencies and models retain th| [FreeToken](https://github.com/FlashML-org/FreeToken) prompt VLM operators, [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) optional weights | [Apache-2.0](freevideo_engine/licenses/FreeToken-Apache-2.0.txt) || [Qwen-Image-2.1](https://huggingface.co/Comfy-Org/Qwen-Image-2.1) Turbo image generation models, including Qwen3-VL-8B and VAE (optional download; [notice](freevideo_engine/image_gen/licenses/NOTICE.txt) installed with the models), © 2026 Hangzhou Tongyi Laboratory Technology Co., Ltd. All Rights Reserved. | [Qwen RESEARCH LICENSE AGREEMENT](https://huggingface.co/Comfy-Org/Qwen-Image-2.1/blob/main/LICENSE) || [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) 1.13.8 speech runtime, built by FreeVideo without text-to-speech, ALSA and PortAudio (modified build; `__init__.py` edited; source snapshot 11afbd00 kept by FreeVideo; optional download for automatic subtitles, [build note](freevideo_engine/licenses/sherpa-onnx-build-NOTE.txt)). Includes statically linked kaldi-native-fbank, kaldi-decoder, kaldifst, OpenFst, simple-sentencepiece (Apache-2.0), KissFFT (BSD-3-Clause), pybind11 (BSD-3-Clause), nlohmann/json (MIT), hclust-cpp and Darts-clone (BSD-2-Clause), ThreadPool (Zlib). The Windows build also contains msvcp140.dll and msvcp140_1.dll (Microsoft Visual C++ 2022 runtime 14.44.35211, © Microsoft Corporation), unmodified and redistributed as permitted by Microsoft for its redistributable files. | Apache-2.0 |+| [CosyVoice](https://github.com/FunAudioLLM/CosyVoice) code adapted for dubbing and [Fun-CosyVoice3-0.5B-2512](https://modelscope.cn/models/FunAudioLLM/Fun-CosyVoice3-0.5B-2512) weights, © 2024 Alibaba Inc (optional download); includes MIT-licensed parts of Matcha-TTS, HiFi-GAN, x_transformers and Snake | [Apache-2.0](freevideo_engine/licenses/CosyVoice-Apache-2.0.txt); [MIT notices](freevideo_engine/licenses/CosyVoice-NOTICE.txt) || [ONNX Runtime](https://github.com/microsoft/onnxruntime) 1.28.2 (CPU), © Microsoft Corporation; its third-party notices are shipped inside the sherpa-onnx wheel | MIT || [Eigen](https://gitlab.com/libeigen/eigen) 5.0.1, compiled into sherpa-onnx and ONNX Runtime; source: https://gitlab.com/libeigen/eigen/-/tree/5.0.1 | MPL-2.0 || [Silero VAD](https://github.com/snakers4/silero-vad) v5.0 voice-activity model, © 2020-present Silero Team (optional download with subtitles, license file downloaded with the model; the onnx-community export is downloaded with the music model) | [MIT](freevideo_engine/licenses/Silero-VAD-MIT.txt) |diff --git a/__init__.py b/__init__.pyindex 3349199..b6214c5 100644--- a/__init__.py+++ b/__init__.py@@ -31,6 +31,12 @@ async def comfy_entrypoint():except Exception as error:import logginglogging.warning('[FreeVideo] Image generation routes not registered: %s', error)+ try:+ from .freevideo_engine.dubbing.service import register as dubbing_routes+ dubbing_routes()+ except Exception as error:+ import logging+ logging.warning('[FreeVideo] Dubbing routes not registered: %s', error)from .freevideo_engine.comfy_launcher_api import register as launcher_routeslauncher_routes()from .freevideo_engine.comfy_updates import register as update_routesdiff --git a/constraints/unified.txt b/constraints/unified.txtindex 882a9d6..b281491 100644--- a/constraints/unified.txt+++ b/constraints/unified.txt@@ -17,6 +17,7 @@ cuda-toolkit==13.0.3.0cryptography==50.0.1einops==0.8.2filelock==3.32.3+flatbuffers==25.12.19fsspec==2026.7.0h11==0.16.0hf-xet==1.6.1a0@@ -50,10 +51,12 @@ nvidia-nvjitlink==13.3.33nvidia-nvshmem-cu13==3.4.5nvidia-nvtx==13.0.85omegaconf==2.3.1+onnxruntime==1.31.1packaging==26.3peft==0.20.0pillow==12.3.0pip==26.2.1+protobuf==7.36.2psutil==7.2.2pygments==2.21.0pycparser==3.0diff --git a/freevideo_engine/dubbing/__init__.py b/freevideo_engine/dubbing/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/freevideo_engine/dubbing/align.py b/freevideo_engine/dubbing/align.pynew file mode 100644index 0000000..5c3ec0d--- /dev/null+++ b/freevideo_engine/dubbing/align.py@@ -0,0 +1,238 @@+"""Fit a synthesized line to a target duration, whatever backend made it.++A backend exposes `synth(speed)` -> (mono float32 samples, sample rate); `speed`+above 1 makes the line shorter. The first pass is natural speed, the second+pass uses the measured ratio, and a third corrects what is still outside the+tolerance. The voiced part (first to last loud 5 ms frame) is then closed to the wanted length by a+pitch-preserving tempo change within `max_stretch`, and placed at the wanted onset in a file of exactly+the target number of samples: silence fills what is short. Voice that cannot fit is never cut: the file+is then longer than the target and reports `overrun` seconds (`fitted` false). A few milliseconds of the quiet+edge around the voiced part are kept, so soft consonants and the decay of the last sound survive.+"""+from fractions import Fraction++import numpy as np++TOLERANCE = 0.02+FIT_SECONDS = 0.04+FADE = 0.003+EDGE = 0.02+MEASURE_FRAME = 0.005+MAX_STRETCH = 1.25+SPEED_RANGE = (0.6, 1.8)+++LEAD_KEEP = 0.05+TRAIL_KEEP = 0.12+SILENCE_LEVEL = 0.02+SPAN_SHARE = 0.02 # a line starts and ends where the smoothed energy crosses this share of its peak+NOISE_MARGIN = 4. # the voiced level is at least this many times the noise floor (10th percentile of the frames)+NOISE_CAP = 0.1 # ... but never above this share of the loudest frame+++def seconds(samples, rate):+ return len(samples) / rate+++def trim_silence(samples, rate, lead_keep=LEAD_KEEP, trail_keep=TRAIL_KEEP, level=SILENCE_LEVEL):+ """Cut the quiet ends a model leaves around a line, keeping a short margin of each.++ The language model sometimes opens with seconds of silence; measuring that as speech would make+ every pass miss the target. Quiet is below `level` times the loudest 10 ms frame.+ """+ samples = np.asarray(samples, dtype=np.float32)+ frame = max(1, rate // 100)+ count = len(samples) // frame+ if count < 2:+ return samples+ rms = np.sqrt((samples[:count * frame].reshape(count, frame) ** 2).mean(axis=1))+ loud = np.nonzero(rms > level * rms.max())[0]+ if not len(loud):+ return samples+ start = max(0, int(loud[0] * frame - lead_keep * rate))+ end = min(len(samples), int((loud[-1] + 1) * frame + trail_keep * rate))+ return samples[start:end]+++def stretch(samples, rate, ratio):+ """Return `samples` played `ratio` times faster (ratio > 1 shortens), pitch kept."""+ import av+ graph = av.filter.Graph()+ source = graph.add_abuffer(format='flt', sample_rate=rate, layout='mono', time_base=Fraction(1, rate))+ tempo = graph.add('atempo', '%.6f' % ratio)+ sink = graph.add('abuffersink')+ source.link_to(tempo)+ tempo.link_to(sink)+ graph.configure()+ frame = av.AudioFrame.from_ndarray(np.ascontiguousarray(samples, dtype=np.float32)[None, :],+ format='flt', layout='mono')+ frame.sample_rate, frame.pts = rate, 0+ graph.push(frame)+ graph.push(None)+ parts = []+ while True:+ try:+ parts.append(sink.pull().to_ndarray()[0])+ except (av.error.EOFError, av.error.BlockingIOError):+ break+ return np.concatenate(parts).astype(np.float32) if parts else np.zeros(0, np.float32)+++def voiced_span(samples, rate, level=SILENCE_LEVEL):+ """(first, end) sample of the voiced part: from the first to the last loud 5 ms frame, or None."""+ samples = np.asarray(samples, dtype=np.float32)+ frame = max(1, int(rate * MEASURE_FRAME))+ count = len(samples) // frame+ if count < 1:+ return None+ rms = np.sqrt((samples[:count * frame].reshape(count, frame) ** 2).mean(axis=1))+ if not rms.max() > 0:+ return None+ # A recording with music or room noise comes back with that noise floor: the level rises above it+ # (never past NOISE_CAP of the loudest frame), and a lone loud frame is not voice.+ threshold = min(max(level * rms.max(), NOISE_MARGIN * np.percentile(rms, 10)), NOISE_CAP * rms.max())+ above = rms > threshold+ joined = above & (np.concatenate([above[1:], [False]]) | np.concatenate([[False], above[:-1]]))+ loud = np.nonzero(joined if joined.any() else above)[0]+ if not len(loud):+ return None+ return int(loud[0] * frame), int((loud[-1] + 1) * frame)+++def energy_span(samples, rate, share=SPAN_SHARE):+ """(first, end) sample where the 5 ms smoothed energy stays above `share` of its peak: the definition+ the timeline of an original line is taken with. None for silence."""+ data = np.asarray(samples, dtype=np.float64)+ window = max(1, int(rate * MEASURE_FRAME))+ if len(data) < window:+ return None+ total = np.concatenate([[0.], np.cumsum(data * data)])+ half = window // 2+ centre = np.arange(len(data))+ low, high = np.clip(centre - half, 0, len(data)), np.clip(centre - half + window, 0, len(data))+ level = np.sqrt((total[high] - total[low]) / window)+ if not level.max() > 0:+ return None+ loud = np.nonzero(level > share * level.max())[0]+ return int(loud[0]), int(loud[-1]) + 1+++def place(voice, rate, total, onset, lead=0, trail=0):+ """`voice` (`lead` quiet samples, the voiced part, `trail` quiet samples) laid into silence of+ round(total*rate) samples so that the voiced part starts at `onset`; returns (samples, overrun).++ `overrun` is the number of voiced samples past the end of the line. The file is then longer by that+ much and nothing is cut; otherwise the quiet tail is trimmed to the exact length. Also returns the+ voiced part's first and end sample in the result: the placement is known, so it is not measured again+ (a line that fills its slot has no silence left, and the measure's noise floor would then be speech)."""+ size = int(round(total * rate))+ voiced = len(voice) - lead - trail+ start = int(round(onset * rate)) - lead+ take = np.array(voice, dtype=np.float32)+ if start < 0:+ take, start, lead = take[-start:], 0, lead + start+ overrun = max(0, start + lead + voiced - size)+ if overrun <= int(MEASURE_FRAME * rate): # under the resolution of the measurement+ overrun = 0+ take = take[:size - start]+ length = max(size, start + len(take))+ first = start + max(lead, 0)+ last = min(first + voiced, start + len(take))+ for width, edge in ((int(FADE * rate), 'in'), (int(FADE * rate), 'out')):+ width = min(width, len(take))+ if width > 1:+ ramp = np.linspace(0, 1, width, dtype=np.float32)+ if edge == 'in':+ take[:width] *= ramp+ else:+ take[-width:] *= ramp[::-1]+ out = np.zeros(length, np.float32)+ out[start:start + len(take)] = take+ return out, overrun, first, last+++def fit(synth, target, *, onset=0., offset=None, tolerance=TOLERANCE, max_stretch=MAX_STRETCH,+ speed_range=SPEED_RANGE, check=lambda: None):+ """Run the passes; return dict(samples, rate, seconds, onset, offset, error, onset_error,+ offset_error, speed, passes, stretch,+ fitted, warning, overrun, natural). `samples` has exactly round(target*rate) samples unless the voice overruns; the voiced part is+ wanted from `onset` to `offset` (default: the whole line)."""+ if not target > 0:+ raise ValueError('Target duration must be positive')+ offset = target if offset is None else offset+ wanted = offset - onset+ if not 0 <= onset < offset <= target + 1e-9:+ raise ValueError('The wanted voiced part must lie inside the line')+ low, high = speed_range+ tried = []++ def run(speed):+ check()+ samples, rate = synth(speed)+ span = voiced_span(samples, rate)+ if span is None:+ raise ValueError('The dubbing backend returned no audio')+ samples = np.asarray(samples, dtype=np.float32)+ edge = int(EDGE * rate)+ voice = samples[span[0]:span[1]]+ # quiet samples around the voiced part, kept as they are (not stretched)+ edges = (samples[max(0, span[0] - edge):span[0]], samples[span[1]:span[1] + edge])+ tried.append((speed, voice, rate, seconds(voice, rate), edges))+ return tried[-1]++ def off(entry):+ return abs(entry[3] - wanted) / wanted++ speed = 1.0+ entry = run(speed)+ while off(entry) > tolerance and len(tried) < 3:+ # the length scales with 1/speed, so the next speed is the measured ratio+ speed = min(max(speed * entry[3] / wanted, low), high)+ if speed == tried[-1][0]:+ break+ entry = run(speed)+ speed, voice, rate, length, edges = min(tried, key=off)+ padded_edges = edges++ def compose(start, width):+ """The voiced part closed to `width` seconds and placed from `start`; (out, overrun, first, last, ratio)."""+ source, ratio, current, length_now = voice, 1.0, voice, length+ for _ in range(2):+ want = min(max(ratio * length_now / width, 1 / max_stretch), max_stretch)+ if abs(want - ratio) < 0.001 and ratio != 1.0 or abs(length_now - width) <= 0.004 and ratio == 1.0:+ break+ ratio = want+ current = stretch(source, rate, ratio)+ length_now = seconds(current, rate)+ padded = np.concatenate([padded_edges[0], current, padded_edges[1]])+ out, overrun, first, last = place(padded, rate, target, start, lead=len(padded_edges[0]),+ trail=len(padded_edges[1]))+ return out, overrun, first, last, ratio++ # The placed file is measured the way the originals' timeline is (smoothed energy above a share of its+ # peak): the tempo change and the quiet edges spread a little energy past the voiced part, so the first+ # placement is corrected by what is measured, twice at most.+ start, width = onset, wanted+ best = None+ for attempt in range(3):+ out, overrun, first, last, ratio = compose(start, width)+ span = energy_span(out, rate)+ first, last = (span[0], span[1]) if span else (first, last)+ first, last = first / rate, last / rate+ miss = max(abs(first - onset), abs(last - offset)) + (overrun > 0)+ if best is None or miss < best[0]:+ best = (miss, out, overrun, first, last, ratio)+ if miss <= 0.004 or attempt == 2:+ break+ start = max(0., start + (onset - first))+ width = max(0.25 * wanted, width + (wanted - (last - first)))+ _, out, overrun, first, last, ratio = best+ error = (last - first) - wanted+ onset_error, offset_error = first - onset, last - offset+ overrun = overrun / rate+ fitted = abs(onset_error) <= FIT_SECONDS and abs(offset_error) <= FIT_SECONDS and not overrun+ warning = None if fitted else 'too_long' if (overrun or offset_error > 0) else 'too_short'+ return dict(samples=out, rate=rate, seconds=len(out) / rate, onset=first, offset=last, error=error,+ onset_error=onset_error, offset_error=offset_error,+ speed=speed, passes=len(tried), stretch=ratio, fitted=fitted, warning=warning, overrun=overrun,+ natural=tried[0][3])diff --git a/freevideo_engine/dubbing/backends.py b/freevideo_engine/dubbing/backends.pynew file mode 100644index 0000000..3ceed52--- /dev/null+++ b/freevideo_engine/dubbing/backends.py@@ -0,0 +1,63 @@+"""Dubbing backends behind one interface, so a model can be replaced without touching the service.++A backend is built from its model folder, says a line in the voice of one reference recording and+frees its memory in `close`. `speed` above 1 makes the line shorter; `align.fit` uses it to reach a+target duration.+"""+import importlib++REGISTRY = {+ 'cosyvoice3': 'freevideo_engine.dubbing.backends:CosyVoice3Backend',+}+DEFAULT = 'cosyvoice3'+MAX_REFERENCE_SECONDS = 29.0+++class Backend:+ name = ''+ sample_rate = 24000+ languages = ()+ # Whether the reference needs a transcript of what it says+ needs_reference_text = True+ supports_instruction = False++ def __init__(self, model_dir, device='cuda'):+ raise NotImplementedError++ def synth(self, text, reference, reference_text, speed=1.0, instruction=None):+ """Return (mono float32 samples, sample rate)."""+ raise NotImplementedError++ def close(self):+ pass+++class CosyVoice3Backend(Backend):+ name = 'cosyvoice3'+ sample_rate = 24000+ languages = ('zh', 'en', 'ja', 'ko', 'de', 'es', 'fr', 'it', 'ru')+ supports_instruction = True++ def __init__(self, model_dir, device='cuda'):+ from .cosyvoice3.build import CosyVoice3+ self.model = CosyVoice3(model_dir, device=device)++ def synth(self, text, reference, reference_text, speed=1.0, instruction=None):+ if instruction:+ samples = self.model.instruct(text, instruction, reference, speed=speed)+ else:+ samples = self.model.zero_shot(text, reference_text, reference, speed=speed)+ return samples, self.sample_rate++ def close(self):+ self.model = None+++def names():+ return sorted(REGISTRY)+++def create(name, model_dir, device='cuda'):+ """Build the named backend; an unknown name raises KeyError."""+ module, _, cls = REGISTRY[name].partition(':')+ return getattr(importlib.import_module(module), cls)(model_dir, device=device)diff --git a/freevideo_engine/dubbing/cosyvoice3/__init__.py b/freevideo_engine/dubbing/cosyvoice3/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/freevideo_engine/dubbing/cosyvoice3/build.py b/freevideo_engine/dubbing/cosyvoice3/build.pynew file mode 100644index 0000000..6e4c6a2--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/build.py@@ -0,0 +1,115 @@+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+"""Build Fun-CosyVoice3-0.5B from its published weights without hyperpyyaml.++The parameters below are those of the model's cosyvoice3.yaml.+"""+import os+from functools import partial++import torch+from omegaconf import DictConfig++from freevideo_engine.dubbing.cosyvoice3.cli.frontend import CosyVoiceFrontEnd+from freevideo_engine.dubbing.cosyvoice3.cli.model import CosyVoice3Model+from freevideo_engine.dubbing.cosyvoice3.flow.DiT.dit import DiT+from freevideo_engine.dubbing.cosyvoice3.flow.flow import CausalMaskedDiffWithDiT+from freevideo_engine.dubbing.cosyvoice3.flow.flow_matching import CausalConditionalCFM+from freevideo_engine.dubbing.cosyvoice3.hifigan.f0_predictor import CausalConvRNNF0Predictor+from freevideo_engine.dubbing.cosyvoice3.hifigan.generator import CausalHiFTGenerator+from freevideo_engine.dubbing.cosyvoice3.llm.llm import CosyVoice3LM, Qwen2Encoder+from freevideo_engine.dubbing.cosyvoice3.tokenizer.tokenizer import get_qwen_tokenizer+from freevideo_engine.dubbing.cosyvoice3.transformer.upsample_encoder import PreLookaheadLayer+from freevideo_engine.dubbing.cosyvoice3.utils.common import ras_sampling+from freevideo_engine.dubbing.cosyvoice3.utils.mel import mel_spectrogram++SAMPLE_RATE = 24000+QWEN_DIR = 'CosyVoice-BlankEN'+PROMPT_PREFIX = 'You are a helpful assistant.<|endofprompt|>'+++def build_networks(qwen_dir):+ llm = CosyVoice3LM(+ llm_input_size=896, llm_output_size=896, speech_token_size=6561,+ llm=Qwen2Encoder(qwen_dir),+ sampling=partial(ras_sampling, top_p=0.8, top_k=25, win_size=10, tau_r=0.1),+ mix_ratio=[5, 15])+ flow = CausalMaskedDiffWithDiT(+ input_size=80, output_size=80, spk_embed_dim=192, output_type='mel', vocab_size=6561,+ input_frame_rate=25, only_mask_loss=True, token_mel_ratio=2, pre_lookahead_len=3,+ pre_lookahead_layer=PreLookaheadLayer(in_channels=80, channels=1024, pre_lookahead_len=3),+ decoder=CausalConditionalCFM(+ in_channels=240, n_spks=1, spk_emb_dim=80,+ cfm_params=DictConfig({'sigma_min': 1e-06, 'solver': 'euler', 't_scheduler': 'cosine',+ 'training_cfg_rate': 0.2, 'inference_cfg_rate': 0.7, 'reg_loss_type': 'l1'}),+ estimator=DiT(dim=1024, depth=22, heads=16, dim_head=64, ff_mult=2, mel_dim=80, mu_dim=80,+ spk_dim=80, out_channels=80, static_chunk_size=50, num_decoding_left_chunks=-1)))+ hift = CausalHiFTGenerator(+ in_channels=80, base_channels=512, nb_harmonics=8, sampling_rate=SAMPLE_RATE,+ nsf_alpha=0.1, nsf_sigma=0.003, nsf_voiced_threshold=10,+ upsample_rates=[8, 5, 3], upsample_kernel_sizes=[16, 11, 7],+ istft_params={'n_fft': 16, 'hop_len': 4},+ resblock_kernel_sizes=[3, 7, 11], resblock_dilation_sizes=[[1, 3, 5]] * 3,+ source_resblock_kernel_sizes=[7, 7, 11], source_resblock_dilation_sizes=[[1, 3, 5]] * 3,+ lrelu_slope=0.1, audio_limit=0.99, conv_pre_look_right=4,+ f0_predictor=CausalConvRNNF0Predictor(num_class=1, in_channels=80, cond_channels=512))+ return llm, flow, hift+++def _feat_extractor():+ return partial(mel_spectrogram, n_fft=1920, num_mels=80, sampling_rate=SAMPLE_RATE, hop_size=480,+ win_size=1920, fmin=0, fmax=None, center=False)+++class CosyVoice3:+ """Zero-shot voice cloning with Fun-CosyVoice3-0.5B; every call returns one float32 numpy waveform."""++ sample_rate = SAMPLE_RATE++ def __init__(self, model_dir, device='cuda', fp16=False):+ qwen_dir = os.path.join(model_dir, QWEN_DIR)+ self.frontend = CosyVoiceFrontEnd(+ lambda: get_qwen_tokenizer(token_path=qwen_dir, skip_special_tokens=True, version='cosyvoice3'),+ _feat_extractor(),+ os.path.join(model_dir, 'campplus.onnx'),+ os.path.join(model_dir, 'speech_tokenizer_v3.onnx'),+ device=device)+ llm, flow, hift = build_networks(qwen_dir)+ self.model = CosyVoice3Model(llm, flow, hift, device, fp16)+ self.model.load(os.path.join(model_dir, 'llm.pt'), os.path.join(model_dir, 'flow.pt'),+ os.path.join(model_dir, 'hift.pt'))++ def _run(self, texts, make_input, speed):+ import numpy as np+ parts = []+ for piece in self.frontend.text_normalize(texts, split=True, text_frontend=True):+ inp = make_input(piece)+ speech = self.model.tts(+ text=inp['text'], flow_embedding=inp['flow_embedding'], llm_embedding=inp['llm_embedding'],+ prompt_text=inp['prompt_text'],+ llm_prompt_speech_token=inp.get('llm_prompt_speech_token', torch.zeros(1, 0, dtype=torch.int32)),+ flow_prompt_speech_token=inp['flow_prompt_speech_token'],+ prompt_speech_feat=inp['prompt_speech_feat'], speed=speed)+ parts.append(speech.squeeze(0).float().cpu().numpy())+ return np.concatenate(parts).astype('float32')++ def zero_shot(self, text, prompt_text, prompt_wav, speed=1.0):+ """Say `text` in the voice of `prompt_wav`, whose transcript is `prompt_text`."""+ prompt = PROMPT_PREFIX + prompt_text+ return self._run(text, lambda t: self.frontend.frontend_zero_shot(t, prompt, prompt_wav, SAMPLE_RATE), speed)++ def instruct(self, text, instruction, prompt_wav, speed=1.0):+ """Say `text` in the voice of `prompt_wav` in the manner `instruction` (a Chinese sentence) describes."""+ prompt = f'You are a helpful assistant. {instruction}<|endofprompt|>'+ return self._run(text, lambda t: self.frontend.frontend_instruct2(t, prompt, prompt_wav, SAMPLE_RATE), speed)diff --git a/freevideo_engine/dubbing/cosyvoice3/cli/__init__.py b/freevideo_engine/dubbing/cosyvoice3/cli/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/freevideo_engine/dubbing/cosyvoice3/cli/frontend.py b/freevideo_engine/dubbing/cosyvoice3/cli/frontend.pynew file mode 100644index 0000000..3781db4--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/cli/frontend.py@@ -0,0 +1,153 @@+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# Modified by FreeVideo: reduced to the zero-shot and instruct2 inputs; onnxruntime, wetext and+# inflect are optional so the package imports without them.+import logging+import re+from functools import partial+from typing import Callable++import numpy as np+import torch+import torchaudio.compliance.kaldi as kaldi++from freevideo_engine.dubbing.cosyvoice3.utils.file_utils import load_wav+from freevideo_engine.dubbing.cosyvoice3.utils.frontend_utils import (+ contains_chinese, is_only_punctuation, remove_bracket, replace_blank, replace_corner_mark,+ spell_out_number, split_paragraph)+from freevideo_engine.dubbing.cosyvoice3.utils.mel import log_mel_spectrogram++logger = logging.getLogger(__name__)+++class CosyVoiceFrontEnd:++ def __init__(self,+ get_tokenizer: Callable,+ feat_extractor: Callable,+ campplus_model: str,+ speech_tokenizer_model: str,+ device='cpu'):+ import onnxruntime+ self.tokenizer = get_tokenizer()+ self.feat_extractor = feat_extractor+ self.device = torch.device(device)+ option = onnxruntime.SessionOptions()+ option.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL+ option.intra_op_num_threads = 1+ # both small networks run on the CPU so they take no video memory next to the three generators+ self.campplus_session = onnxruntime.InferenceSession(+ campplus_model, sess_options=option, providers=['CPUExecutionProvider'])+ self.speech_tokenizer_session = onnxruntime.InferenceSession(+ speech_tokenizer_model, sess_options=option, providers=['CPUExecutionProvider'])+ self.inflect_parser = None+ try:+ import inflect+ self.inflect_parser = inflect.engine()+ except Exception:+ logger.info('inflect is not available, digits in English text are kept as written')+ self.zh_tn_model = self.en_tn_model = None+ try:+ from wetext import Normalizer+ self.zh_tn_model = Normalizer(remove_erhua=False)+ self.en_tn_model = Normalizer()+ except Exception:+ logger.info('wetext is not available, text is not normalized')++ def _extract_text_token(self, text):+ text_token = self.tokenizer.encode(text)+ text_token = torch.tensor([text_token], dtype=torch.int32).to(self.device)+ text_token_len = torch.tensor([text_token.shape[1]], dtype=torch.int32).to(self.device)+ return text_token, text_token_len++ def _extract_speech_token(self, prompt_wav):+ speech = load_wav(prompt_wav, 16000)+ assert speech.shape[1] / 16000 <= 30, 'do not support extract speech token for audio longer than 30s'+ feat = log_mel_spectrogram(speech, n_mels=128)+ speech_token = self.speech_tokenizer_session.run(+ None, {self.speech_tokenizer_session.get_inputs()[0].name: feat.detach().cpu().numpy(),+ self.speech_tokenizer_session.get_inputs()[1].name: np.array([feat.shape[2]], dtype=np.int32)}+ )[0].flatten().tolist()+ speech_token = torch.tensor([speech_token], dtype=torch.int32).to(self.device)+ speech_token_len = torch.tensor([speech_token.shape[1]], dtype=torch.int32).to(self.device)+ return speech_token, speech_token_len++ def _extract_spk_embedding(self, prompt_wav):+ speech = load_wav(prompt_wav, 16000)+ feat = kaldi.fbank(speech, num_mel_bins=80, dither=0, sample_frequency=16000)+ feat = feat - feat.mean(dim=0, keepdim=True)+ embedding = self.campplus_session.run(+ None, {self.campplus_session.get_inputs()[0].name: feat.unsqueeze(dim=0).cpu().numpy()}+ )[0].flatten().tolist()+ return torch.tensor([embedding]).to(self.device)++ def _extract_speech_feat(self, prompt_wav):+ speech = load_wav(prompt_wav, 24000)+ speech_feat = self.feat_extractor(speech).squeeze(dim=0).transpose(0, 1).to(self.device)+ speech_feat = speech_feat.unsqueeze(dim=0)+ speech_feat_len = torch.tensor([speech_feat.shape[1]], dtype=torch.int32).to(self.device)+ return speech_feat, speech_feat_len++ def text_normalize(self, text, split=True, text_frontend=True):+ # NOTE skip the text frontend when a control symbol is in the text+ if '<|' in text and '|>' in text:+ text_frontend = False+ if text_frontend is False or text == '':+ return [text] if split is True else text+ text = text.strip()+ if contains_chinese(text):+ if self.zh_tn_model is not None:+ text = self.zh_tn_model.normalize(text)+ text = text.replace("\n", "")+ text = replace_blank(text)+ text = replace_corner_mark(text)+ text = text.replace(".", "。")+ text = text.replace(" - ", ",")+ text = remove_bracket(text)+ text = re.sub(r'[,,、]+$', '。', text)+ lang = 'zh'+ else:+ if self.en_tn_model is not None:+ text = self.en_tn_model.normalize(text)+ if self.inflect_parser is not None:+ text = spell_out_number(text, self.inflect_parser)+ lang = 'en'+ texts = list(split_paragraph(text, self.tokenizer.encode, lang, token_max_n=80,+ token_min_n=60, merge_len=20, comma_split=False))+ texts = [i for i in texts if not is_only_punctuation(i)]+ return texts if split is True else text++ def frontend_zero_shot(self, tts_text, prompt_text, prompt_wav, resample_rate=24000):+ tts_text_token, tts_text_token_len = self._extract_text_token(tts_text)+ prompt_text_token, prompt_text_token_len = self._extract_text_token(prompt_text)+ speech_feat, speech_feat_len = self._extract_speech_feat(prompt_wav)+ speech_token, speech_token_len = self._extract_speech_token(prompt_wav)+ if resample_rate == 24000:+ # force speech_feat % speech_token = 2+ token_len = min(int(speech_feat.shape[1] / 2), speech_token.shape[1])+ speech_feat, speech_feat_len[:] = speech_feat[:, :2 * token_len], 2 * token_len+ speech_token, speech_token_len[:] = speech_token[:, :token_len], token_len+ embedding = self._extract_spk_embedding(prompt_wav)+ return {'prompt_text': prompt_text_token, 'prompt_text_len': prompt_text_token_len,+ 'llm_prompt_speech_token': speech_token, 'llm_prompt_speech_token_len': speech_token_len,+ 'flow_prompt_speech_token': speech_token, 'flow_prompt_speech_token_len': speech_token_len,+ 'prompt_speech_feat': speech_feat, 'prompt_speech_feat_len': speech_feat_len,+ 'llm_embedding': embedding, 'flow_embedding': embedding,+ 'text': tts_text_token, 'text_len': tts_text_token_len}++ def frontend_instruct2(self, tts_text, instruct_text, prompt_wav, resample_rate=24000):+ model_input = self.frontend_zero_shot(tts_text, instruct_text, prompt_wav, resample_rate)+ del model_input['llm_prompt_speech_token']+ del model_input['llm_prompt_speech_token_len']+ return model_inputdiff --git a/freevideo_engine/dubbing/cosyvoice3/cli/model.py b/freevideo_engine/dubbing/cosyvoice3/cli/model.pynew file mode 100644index 0000000..b3ee887--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/cli/model.py@@ -0,0 +1,88 @@+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu)+# 2025 Alibaba Inc (authors: Xiang Lyu, Bofan Zhou)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+#+# Modified by FreeVideo: reduced to the non-streaming CosyVoice3 inference path+# (no streaming, TensorRT, vLLM or JIT variants); the language model runs on the+# calling thread so errors propagate.+import torch+from torch.nn import functional as F+++class CosyVoice3Model:++ def __init__(self, llm: torch.nn.Module, flow: torch.nn.Module, hift: torch.nn.Module, device, fp16: bool = False):+ self.device = torch.device(device)+ self.llm = llm+ self.flow = flow+ self.hift = hift+ self.fp16 = fp16+ # FSQ silent and breath token+ self.silent_tokens = [1, 2, 28, 29, 55, 248, 494, 2241, 2242, 2322, 2323]++ def load(self, llm_model, flow_model, hift_model):+ """Load the three state dicts onto the CPU first and then move each part to the device."""+ for part, path in ((self.llm, llm_model), (self.flow, flow_model)):+ part.load_state_dict(torch.load(path, map_location='cpu', weights_only=True), strict=True)+ hift_state = torch.load(hift_model, map_location='cpu', weights_only=True)+ self.hift.load_state_dict({k.replace('generator.', ''): v for k, v in hift_state.items()}, strict=True)+ for part in (self.llm, self.flow, self.hift):+ part.to(self.device).eval()++ def speech_tokens(self, text, prompt_text, llm_prompt_speech_token, llm_embedding):+ """Run the language model and return the generated speech tokens as a list."""+ device = self.device+ tokens, silent_run, max_silent_run = [], 0, 5+ generator = self.llm.inference(+ text=text.to(device),+ text_len=torch.tensor([text.shape[1]], dtype=torch.int32).to(device),+ prompt_text=prompt_text.to(device),+ prompt_text_len=torch.tensor([prompt_text.shape[1]], dtype=torch.int32).to(device),+ prompt_speech_token=llm_prompt_speech_token.to(device),+ prompt_speech_token_len=torch.tensor([llm_prompt_speech_token.shape[1]], dtype=torch.int32).to(device),+ embedding=llm_embedding.to(device),+ uuid='freevideo')+ for token in generator:+ if token in self.silent_tokens:+ silent_run += 1+ if silent_run > max_silent_run:+ continue+ else:+ silent_run = 0+ tokens.append(token)+ return tokens++ def token2wav(self, token, prompt_token, prompt_feat, embedding, speed=1.0):+ device = self.device+ with torch.autocast(device_type=device.type, enabled=bool(self.fp16) and device.type == 'cuda'):+ mel, _ = self.flow.inference(token=token.to(device, dtype=torch.int32),+ token_len=torch.tensor([token.shape[1]], dtype=torch.int32).to(device),+ prompt_token=prompt_token.to(device),+ prompt_token_len=torch.tensor([prompt_token.shape[1]], dtype=torch.int32).to(device),+ prompt_feat=prompt_feat.to(device),+ prompt_feat_len=torch.tensor([prompt_feat.shape[1]], dtype=torch.int32).to(device),+ embedding=embedding.to(device),+ streaming=False,+ finalize=True)+ if speed != 1.0:+ mel = F.interpolate(mel, size=int(mel.shape[2] / speed), mode='linear')+ speech, _ = self.hift.inference(speech_feat=mel, finalize=True)+ return speech++ def tts(self, text, flow_embedding, llm_embedding, prompt_text, llm_prompt_speech_token,+ flow_prompt_speech_token, prompt_speech_feat, speed=1.0):+ tokens = self.speech_tokens(text, prompt_text, llm_prompt_speech_token, llm_embedding)+ speech = self.token2wav(torch.tensor(tokens).unsqueeze(dim=0), flow_prompt_speech_token,+ prompt_speech_feat, flow_embedding, speed=speed)+ return speech.cpu()diff --git a/freevideo_engine/dubbing/cosyvoice3/flow/DiT/__init__.py b/freevideo_engine/dubbing/cosyvoice3/flow/DiT/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/freevideo_engine/dubbing/cosyvoice3/flow/DiT/dit.py b/freevideo_engine/dubbing/cosyvoice3/flow/DiT/dit.pynew file mode 100644index 0000000..e12fcee--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/flow/DiT/dit.py@@ -0,0 +1,176 @@++"""+ein notation:+b - batch+n - sequence+nt - text sequence+nw - raw wave length+d - dimension+"""++from __future__ import annotations++import torch+from torch import nn+import torch.nn.functional as F+from einops import repeat+from freevideo_engine.dubbing.cosyvoice3.utils.mask import add_optional_chunk_mask+from freevideo_engine.dubbing.cosyvoice3.flow.DiT.modules import (+ RotaryEmbedding,+ TimestepEmbedding,+ ConvNeXtV2Block,+ CausalConvPositionEmbedding,+ DiTBlock,+ AdaLayerNormZero_Final,+ precompute_freqs_cis,+ get_pos_embed_indices,+)+++# Text embedding+++class TextEmbedding(nn.Module):+ def __init__(self, text_num_embeds, text_dim, conv_layers=0, conv_mult=2):+ super().__init__()+ self.text_embed = nn.Embedding(text_num_embeds + 1, text_dim) # use 0 as filler token++ if conv_layers > 0:+ self.extra_modeling = True+ self.precompute_max_pos = 4096 # ~44s of 24khz audio+ self.register_buffer("freqs_cis", precompute_freqs_cis(text_dim, self.precompute_max_pos), persistent=False)+ self.text_blocks = nn.Sequential(+ *[ConvNeXtV2Block(text_dim, text_dim * conv_mult) for _ in range(conv_layers)]+ )+ else:+ self.extra_modeling = False++ def forward(self, text: int["b nt"], seq_len, drop_text=False): # noqa: F722+ batch, text_len = text.shape[0], text.shape[1]+ text = text + 1 # use 0 as filler token. preprocess of batch pad -1, see list_str_to_idx()+ text = text[:, :seq_len] # curtail if character tokens are more than the mel spec tokens+ text = F.pad(text, (0, seq_len - text_len), value=0)++ if drop_text: # cfg for text+ text = torch.zeros_like(text)++ text = self.text_embed(text) # b n -> b n d++ # possible extra modeling+ if self.extra_modeling:+ # sinus pos emb+ batch_start = torch.zeros((batch,), dtype=torch.long)+ pos_idx = get_pos_embed_indices(batch_start, seq_len, max_pos=self.precompute_max_pos)+ text_pos_embed = self.freqs_cis[pos_idx]+ text = text + text_pos_embed++ # convnextv2 blocks+ text = self.text_blocks(text)++ return text+++# noised input audio and context mixing embedding+++class InputEmbedding(nn.Module):+ def __init__(self, mel_dim, text_dim, out_dim, spk_dim=None):+ super().__init__()+ spk_dim = 0 if spk_dim is None else spk_dim+ self.spk_dim = spk_dim+ self.proj = nn.Linear(mel_dim * 2 + text_dim + spk_dim, out_dim)+ self.conv_pos_embed = CausalConvPositionEmbedding(dim=out_dim)++ def forward(+ self,+ x: float["b n d"],+ cond: float["b n d"],+ text_embed: float["b n d"],+ spks: float["b d"],+ ):+ to_cat = [x, cond, text_embed]+ if self.spk_dim > 0:+ spks = repeat(spks, "b c -> b t c", t=x.shape[1])+ to_cat.append(spks)++ x = self.proj(torch.cat(to_cat, dim=-1))+ x = self.conv_pos_embed(x) + x+ return x+++# Transformer backbone using DiT blocks+++class DiT(nn.Module):+ def __init__(+ self,+ *,+ dim,+ depth=8,+ heads=8,+ dim_head=64,+ dropout=0.1,+ ff_mult=4,+ mel_dim=80,+ mu_dim=None,+ long_skip_connection=False,+ spk_dim=None,+ out_channels=None,+ static_chunk_size=50,+ num_decoding_left_chunks=2+ ):+ super().__init__()++ self.time_embed = TimestepEmbedding(dim)+ if mu_dim is None:+ mu_dim = mel_dim+ self.input_embed = InputEmbedding(mel_dim, mu_dim, dim, spk_dim)++ self.rotary_embed = RotaryEmbedding(dim_head)++ self.dim = dim+ self.depth = depth++ self.transformer_blocks = nn.ModuleList(+ [DiTBlock(dim=dim, heads=heads, dim_head=dim_head, ff_mult=ff_mult, dropout=dropout) for _ in range(depth)]+ )+ self.long_skip_connection = nn.Linear(dim * 2, dim, bias=False) if long_skip_connection else None++ self.norm_out = AdaLayerNormZero_Final(dim) # final modulation+ self.proj_out = nn.Linear(dim, mel_dim)+ self.out_channels = out_channels+ self.static_chunk_size = static_chunk_size+ self.num_decoding_left_chunks = num_decoding_left_chunks++ def forward(self, x, mask, mu, t, spks=None, cond=None, streaming=False):+ x = x.transpose(1, 2)+ mu = mu.transpose(1, 2)+ cond = cond.transpose(1, 2)+ spks = spks.unsqueeze(dim=1)+ batch, seq_len = x.shape[0], x.shape[1]+ if t.ndim == 0:+ t = t.repeat(batch)++ # t: conditioning time, c: context (text + masked cond audio), x: noised input audio+ t = self.time_embed(t)+ x = self.input_embed(x, cond, mu, spks.squeeze(1))++ rope = self.rotary_embed.forward_from_seq_len(seq_len)++ if self.long_skip_connection is not None:+ residual = x++ if streaming is True:+ attn_mask = add_optional_chunk_mask(x, mask.bool(), False, False, 0, self.static_chunk_size, -1).unsqueeze(dim=1)+ else:+ attn_mask = add_optional_chunk_mask(x, mask.bool(), False, False, 0, 0, -1).repeat(1, x.size(1), 1).unsqueeze(dim=1)++ for block in self.transformer_blocks:+ x = block(x, t, mask=attn_mask.bool(), rope=rope)++ if self.long_skip_connection is not None:+ x = self.long_skip_connection(torch.cat((x, residual), dim=-1))++ x = self.norm_out(x, t)+ output = self.proj_out(x).transpose(1, 2)+ return outputdiff --git a/freevideo_engine/dubbing/cosyvoice3/flow/DiT/modules.py b/freevideo_engine/dubbing/cosyvoice3/flow/DiT/modules.pynew file mode 100644index 0000000..1b1cd86--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/flow/DiT/modules.py@@ -0,0 +1,653 @@++"""+ein notation:+b - batch+n - sequence+nt - text sequence+nw - raw wave length+d - dimension+"""++from __future__ import annotations+from typing import Optional+import math++import torch+from torch import nn+import torch.nn.functional as F+import torchaudio++++# raw wav to mel spec+class MelSpec(nn.Module):+ def __init__(+ self,+ filter_length=1024,+ hop_length=256,+ win_length=1024,+ n_mel_channels=100,+ target_sample_rate=24_000,+ normalize=False,+ power=1,+ norm=None,+ center=True,+ ):+ super().__init__()+ self.n_mel_channels = n_mel_channels++ self.mel_stft = torchaudio.transforms.MelSpectrogram(+ sample_rate=target_sample_rate,+ n_fft=filter_length,+ win_length=win_length,+ hop_length=hop_length,+ n_mels=n_mel_channels,+ power=power,+ center=center,+ normalized=normalize,+ norm=norm,+ )++ self.register_buffer("dummy", torch.tensor(0), persistent=False)++ def forward(self, inp):+ if len(inp.shape) == 3:+ inp = inp.squeeze(1) # 'b 1 nw -> b nw'++ assert len(inp.shape) == 2++ if self.dummy.device != inp.device:+ self.to(inp.device)++ mel = self.mel_stft(inp)+ mel = mel.clamp(min=1e-5).log()+ return mel+++# sinusoidal position embedding+++class SinusPositionEmbedding(nn.Module):+ def __init__(self, dim):+ super().__init__()+ self.dim = dim++ def forward(self, x, scale=1000):+ device = x.device+ half_dim = self.dim // 2+ emb = math.log(10000) / (half_dim - 1)+ emb = torch.exp(torch.arange(half_dim, device=device).float() * -emb)+ emb = scale * x.unsqueeze(1) * emb.unsqueeze(0)+ emb = torch.cat((emb.sin(), emb.cos()), dim=-1)+ return emb+++# convolutional position embedding+++class ConvPositionEmbedding(nn.Module):+ def __init__(self, dim, kernel_size=31, groups=16):+ super().__init__()+ assert kernel_size % 2 != 0+ self.conv1d = nn.Sequential(+ nn.Conv1d(dim, dim, kernel_size, groups=groups, padding=kernel_size // 2),+ nn.Mish(),+ nn.Conv1d(dim, dim, kernel_size, groups=groups, padding=kernel_size // 2),+ nn.Mish(),+ )++ def forward(self, x: float["b n d"], mask: bool["b n"] | None = None): # noqa: F722+ if mask is not None:+ mask = mask[..., None]+ x = x.masked_fill(~mask, 0.0)++ x = x.permute(0, 2, 1)+ x = self.conv1d(x)+ out = x.permute(0, 2, 1)++ if mask is not None:+ out = out.masked_fill(~mask, 0.0)++ return out+++class CausalConvPositionEmbedding(nn.Module):+ def __init__(self, dim, kernel_size=31, groups=16):+ super().__init__()+ assert kernel_size % 2 != 0+ self.kernel_size = kernel_size+ self.conv1 = nn.Sequential(+ nn.Conv1d(dim, dim, kernel_size, groups=groups, padding=0),+ nn.Mish(),+ )+ self.conv2 = nn.Sequential(+ nn.Conv1d(dim, dim, kernel_size, groups=groups, padding=0),+ nn.Mish(),+ )++ def forward(self, x: float["b n d"], mask: bool["b n"] | None = None): # noqa: F722+ if mask is not None:+ mask = mask[..., None]+ x = x.masked_fill(~mask, 0.0)++ x = x.permute(0, 2, 1)+ x = F.pad(x, (self.kernel_size - 1, 0, 0, 0))+ x = self.conv1(x)+ x = F.pad(x, (self.kernel_size - 1, 0, 0, 0))+ x = self.conv2(x)+ out = x.permute(0, 2, 1)++ if mask is not None:+ out = out.masked_fill(~mask, 0.0)++ return out+++# rotary positional embedding, re-implemented from x_transformers (MIT) so the package is not a dependency+++class RotaryEmbedding(nn.Module):+ def __init__(self, dim, base=10000):+ super().__init__()+ inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float() / dim))+ self.register_buffer("inv_freq", inv_freq)++ def forward_from_seq_len(self, seq_len):+ t = torch.arange(seq_len, device=self.inv_freq.device)+ with torch.autocast(t.device.type, enabled=False):+ t = t.unsqueeze(0)+ freqs = torch.einsum("b i , j -> b i j", t.type_as(self.inv_freq), self.inv_freq)+ freqs = torch.stack((freqs, freqs), dim=-1)+ freqs = freqs.reshape(*freqs.shape[:-2], -1)+ return freqs, 1.0+++def rotate_half(x):+ x = x.reshape(*x.shape[:-1], -1, 2)+ x1, x2 = x.unbind(dim=-1)+ x = torch.stack((-x2, x1), dim=-1)+ return x.reshape(*x.shape[:-2], -1)+++def apply_rotary_pos_emb(t, freqs, scale=1):+ rot_dim, seq_len, orig_dtype = freqs.shape[-1], t.shape[-2], t.dtype+ with torch.autocast(t.device.type, enabled=False):+ freqs = freqs[:, -seq_len:, :]+ if t.ndim == 4 and freqs.ndim == 3:+ freqs = freqs.unsqueeze(1)+ t, t_unrotated = t[..., :rot_dim], t[..., rot_dim:]+ t = (t * freqs.cos() * scale) + (rotate_half(t) * freqs.sin() * scale)+ out = torch.cat((t, t_unrotated), dim=-1)+ return out.type(orig_dtype)+++# rotary positional embedding related+++def precompute_freqs_cis(dim: int, end: int, theta: float = 10000.0, theta_rescale_factor=1.0):+ # proposed by reddit user bloc97, to rescale rotary embeddings to longer sequence length without fine-tuning+ # has some connection to NTK literature+ # https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/+ # https://github.com/lucidrains/rotary-embedding-torch/blob/main/rotary_embedding_torch/rotary_embedding_torch.py+ theta *= theta_rescale_factor ** (dim / (dim - 2))+ freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim))+ t = torch.arange(end, device=freqs.device) # type: ignore+ freqs = torch.outer(t, freqs).float() # type: ignore+ freqs_cos = torch.cos(freqs) # real part+ freqs_sin = torch.sin(freqs) # imaginary part+ return torch.cat([freqs_cos, freqs_sin], dim=-1)+++def get_pos_embed_indices(start, length, max_pos, scale=1.0):+ # length = length if isinstance(length, int) else length.max()+ scale = scale * torch.ones_like(start, dtype=torch.float32) # in case scale is a scalar+ pos = (+ start.unsqueeze(1)+ + (torch.arange(length, device=start.device, dtype=torch.float32).unsqueeze(0) * scale.unsqueeze(1)).long()+ )+ # avoid extra long error.+ pos = torch.where(pos < max_pos, pos, max_pos - 1)+ return pos+++# Global Response Normalization layer (Instance Normalization ?)+++class GRN(nn.Module):+ def __init__(self, dim):+ super().__init__()+ self.gamma = nn.Parameter(torch.zeros(1, 1, dim))+ self.beta = nn.Parameter(torch.zeros(1, 1, dim))++ def forward(self, x):+ Gx = torch.norm(x, p=2, dim=1, keepdim=True)+ Nx = Gx / (Gx.mean(dim=-1, keepdim=True) + 1e-6)+ return self.gamma * (x * Nx) + self.beta + x+++# ConvNeXt-V2 Block https://github.com/facebookresearch/ConvNeXt-V2/blob/main/models/convnextv2.py+# ref: https://github.com/bfs18/e2_tts/blob/main/rfwave/modules.py#L108+++class ConvNeXtV2Block(nn.Module):+ def __init__(+ self,+ dim: int,+ intermediate_dim: int,+ dilation: int = 1,+ ):+ super().__init__()+ padding = (dilation * (7 - 1)) // 2+ self.dwconv = nn.Conv1d(+ dim, dim, kernel_size=7, padding=padding, groups=dim, dilation=dilation+ ) # depthwise conv+ self.norm = nn.LayerNorm(dim, eps=1e-6)+ self.pwconv1 = nn.Linear(dim, intermediate_dim) # pointwise/1x1 convs, implemented with linear layers+ self.act = nn.GELU()+ self.grn = GRN(intermediate_dim)+ self.pwconv2 = nn.Linear(intermediate_dim, dim)++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ residual = x+ x = x.transpose(1, 2) # b n d -> b d n+ x = self.dwconv(x)+ x = x.transpose(1, 2) # b d n -> b n d+ x = self.norm(x)+ x = self.pwconv1(x)+ x = self.act(x)+ x = self.grn(x)+ x = self.pwconv2(x)+ return residual + x+++# AdaLayerNormZero+# return with modulated x for attn input, and params for later mlp modulation+++class AdaLayerNormZero(nn.Module):+ def __init__(self, dim):+ super().__init__()++ self.silu = nn.SiLU()+ self.linear = nn.Linear(dim, dim * 6)++ self.norm = nn.LayerNorm(dim, elementwise_affine=False, eps=1e-6)++ def forward(self, x, emb=None):+ emb = self.linear(self.silu(emb))+ shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = torch.chunk(emb, 6, dim=1)++ x = self.norm(x) * (1 + scale_msa[:, None]) + shift_msa[:, None]+ return x, gate_msa, shift_mlp, scale_mlp, gate_mlp+++# AdaLayerNormZero for final layer+# return only with modulated x for attn input, cuz no more mlp modulation+++class AdaLayerNormZero_Final(nn.Module):+ def __init__(self, dim):+ super().__init__()++ self.silu = nn.SiLU()+ self.linear = nn.Linear(dim, dim * 2)++ self.norm = nn.LayerNorm(dim, elementwise_affine=False, eps=1e-6)++ def forward(self, x, emb):+ emb = self.linear(self.silu(emb))+ scale, shift = torch.chunk(emb, 2, dim=1)++ x = self.norm(x) * (1 + scale)[:, None, :] + shift[:, None, :]+ return x+++# FeedForward+++class FeedForward(nn.Module):+ def __init__(self, dim, dim_out=None, mult=4, dropout=0.0, approximate: str = "none"):+ super().__init__()+ inner_dim = int(dim * mult)+ dim_out = dim_out if dim_out is not None else dim++ activation = nn.GELU(approximate=approximate)+ project_in = nn.Sequential(nn.Linear(dim, inner_dim), activation)+ self.ff = nn.Sequential(project_in, nn.Dropout(dropout), nn.Linear(inner_dim, dim_out))++ def forward(self, x):+ return self.ff(x)+++# Attention with possible joint part+# modified from diffusers/src/diffusers/models/attention_processor.py+++class Attention(nn.Module):+ def __init__(+ self,+ processor: JointAttnProcessor | AttnProcessor,+ dim: int,+ heads: int = 8,+ dim_head: int = 64,+ dropout: float = 0.0,+ context_dim: Optional[int] = None, # if not None -> joint attention+ context_pre_only=None,+ ):+ super().__init__()++ if not hasattr(F, "scaled_dot_product_attention"):+ raise ImportError("Attention equires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.")++ self.processor = processor++ self.dim = dim+ self.heads = heads+ self.inner_dim = dim_head * heads+ self.dropout = dropout++ self.context_dim = context_dim+ self.context_pre_only = context_pre_only++ self.to_q = nn.Linear(dim, self.inner_dim)+ self.to_k = nn.Linear(dim, self.inner_dim)+ self.to_v = nn.Linear(dim, self.inner_dim)++ if self.context_dim is not None:+ self.to_k_c = nn.Linear(context_dim, self.inner_dim)+ self.to_v_c = nn.Linear(context_dim, self.inner_dim)+ if self.context_pre_only is not None:+ self.to_q_c = nn.Linear(context_dim, self.inner_dim)++ self.to_out = nn.ModuleList([])+ self.to_out.append(nn.Linear(self.inner_dim, dim))+ self.to_out.append(nn.Dropout(dropout))++ if self.context_pre_only is not None and not self.context_pre_only:+ self.to_out_c = nn.Linear(self.inner_dim, dim)++ def forward(+ self,+ x: float["b n d"], # noised input x # noqa: F722+ c: float["b n d"] = None, # context c # noqa: F722+ mask: bool["b n"] | None = None, # noqa: F722+ rope=None, # rotary position embedding for x+ c_rope=None, # rotary position embedding for c+ ) -> torch.Tensor:+ if c is not None:+ return self.processor(self, x, c=c, mask=mask, rope=rope, c_rope=c_rope)+ else:+ return self.processor(self, x, mask=mask, rope=rope)+++# Attention processor+++class AttnProcessor:+ def __init__(self):+ pass++ def __call__(+ self,+ attn: Attention,+ x: float["b n d"], # noised input x # noqa: F722+ mask: bool["b n"] | None = None, # noqa: F722+ rope=None, # rotary position embedding+ ) -> torch.FloatTensor:+ batch_size = x.shape[0]++ # `sample` projections.+ query = attn.to_q(x)+ key = attn.to_k(x)+ value = attn.to_v(x)++ # apply rotary position embedding+ if rope is not None:+ freqs, xpos_scale = rope+ q_xpos_scale, k_xpos_scale = (xpos_scale, xpos_scale**-1.0) if xpos_scale is not None else (1.0, 1.0)++ query = apply_rotary_pos_emb(query, freqs, q_xpos_scale)+ key = apply_rotary_pos_emb(key, freqs, k_xpos_scale)++ # attention+ inner_dim = key.shape[-1]+ head_dim = inner_dim // attn.heads+ query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)+ key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)+ value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)++ # mask. e.g. inference got a batch with different target durations, mask out the padding+ if mask is not None:+ attn_mask = mask+ if attn_mask.dim() == 2:+ attn_mask = attn_mask.unsqueeze(1).unsqueeze(1) # 'b n -> b 1 1 n'+ attn_mask = attn_mask.expand(batch_size, attn.heads, query.shape[-2], key.shape[-2])+ else:+ attn_mask = None++ x = F.scaled_dot_product_attention(query, key, value, attn_mask=attn_mask, dropout_p=0.0, is_causal=False)+ x = x.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)+ x = x.to(query.dtype)++ # linear proj+ x = attn.to_out[0](x)+ # dropout+ x = attn.to_out[1](x)++ if mask is not None:+ if mask.dim() == 2:+ mask = mask.unsqueeze(-1)+ else:+ mask = mask[:, 0, -1].unsqueeze(-1)+ x = x.masked_fill(~mask, 0.0)++ return x+++# Joint Attention processor for MM-DiT+# modified from diffusers/src/diffusers/models/attention_processor.py+++class JointAttnProcessor:+ def __init__(self):+ pass++ def __call__(+ self,+ attn: Attention,+ x: float["b n d"], # noised input x # noqa: F722+ c: float["b nt d"] = None, # context c, here text # noqa: F722+ mask: bool["b n"] | None = None, # noqa: F722+ rope=None, # rotary position embedding for x+ c_rope=None, # rotary position embedding for c+ ) -> torch.FloatTensor:+ residual = x++ batch_size = c.shape[0]++ # `sample` projections.+ query = attn.to_q(x)+ key = attn.to_k(x)+ value = attn.to_v(x)++ # `context` projections.+ c_query = attn.to_q_c(c)+ c_key = attn.to_k_c(c)+ c_value = attn.to_v_c(c)++ # apply rope for context and noised input independently+ if rope is not None:+ freqs, xpos_scale = rope+ q_xpos_scale, k_xpos_scale = (xpos_scale, xpos_scale**-1.0) if xpos_scale is not None else (1.0, 1.0)+ query = apply_rotary_pos_emb(query, freqs, q_xpos_scale)+ key = apply_rotary_pos_emb(key, freqs, k_xpos_scale)+ if c_rope is not None:+ freqs, xpos_scale = c_rope+ q_xpos_scale, k_xpos_scale = (xpos_scale, xpos_scale**-1.0) if xpos_scale is not None else (1.0, 1.0)+ c_query = apply_rotary_pos_emb(c_query, freqs, q_xpos_scale)+ c_key = apply_rotary_pos_emb(c_key, freqs, k_xpos_scale)++ # attention+ query = torch.cat([query, c_query], dim=1)+ key = torch.cat([key, c_key], dim=1)+ value = torch.cat([value, c_value], dim=1)++ inner_dim = key.shape[-1]+ head_dim = inner_dim // attn.heads+ query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)+ key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)+ value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)++ # mask. e.g. inference got a batch with different target durations, mask out the padding+ if mask is not None:+ attn_mask = F.pad(mask, (0, c.shape[1]), value=True) # no mask for c (text)+ attn_mask = attn_mask.unsqueeze(1).unsqueeze(1) # 'b n -> b 1 1 n'+ attn_mask = attn_mask.expand(batch_size, attn.heads, query.shape[-2], key.shape[-2])+ else:+ attn_mask = None++ x = F.scaled_dot_product_attention(query, key, value, attn_mask=attn_mask, dropout_p=0.0, is_causal=False)+ x = x.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)+ x = x.to(query.dtype)++ # Split the attention outputs.+ x, c = (+ x[:, : residual.shape[1]],+ x[:, residual.shape[1]:],+ )++ # linear proj+ x = attn.to_out[0](x)+ # dropout+ x = attn.to_out[1](x)+ if not attn.context_pre_only:+ c = attn.to_out_c(c)++ if mask is not None:+ mask = mask.unsqueeze(-1)+ x = x.masked_fill(~mask, 0.0)+ # c = c.masked_fill(~mask, 0.) # no mask for c (text)++ return x, c+++# DiT Block+++class DiTBlock(nn.Module):+ def __init__(self, dim, heads, dim_head, ff_mult=4, dropout=0.1):+ super().__init__()++ self.attn_norm = AdaLayerNormZero(dim)+ self.attn = Attention(+ processor=AttnProcessor(),+ dim=dim,+ heads=heads,+ dim_head=dim_head,+ dropout=dropout,+ )++ self.ff_norm = nn.LayerNorm(dim, elementwise_affine=False, eps=1e-6)+ self.ff = FeedForward(dim=dim, mult=ff_mult, dropout=dropout, approximate="tanh")++ def forward(self, x, t, mask=None, rope=None): # x: noised input, t: time embedding+ # pre-norm & modulation for attention input+ norm, gate_msa, shift_mlp, scale_mlp, gate_mlp = self.attn_norm(x, emb=t)++ # attention+ attn_output = self.attn(x=norm, mask=mask, rope=rope)++ # process attention output for input x+ x = x + gate_msa.unsqueeze(1) * attn_output++ ff_norm = self.ff_norm(x) * (1 + scale_mlp[:, None]) + shift_mlp[:, None]+ ff_output = self.ff(ff_norm)+ x = x + gate_mlp.unsqueeze(1) * ff_output++ return x+++# MMDiT Block https://arxiv.org/abs/2403.03206+++class MMDiTBlock(nn.Module):+ r"""+ modified from diffusers/src/diffusers/models/attention.py++ notes.+ _c: context related. text, cond, etc. (left part in sd3 fig2.b)+ _x: noised input related. (right part)+ context_pre_only: last layer only do prenorm + modulation cuz no more ffn+ """++ def __init__(self, dim, heads, dim_head, ff_mult=4, dropout=0.1, context_pre_only=False):+ super().__init__()++ self.context_pre_only = context_pre_only++ self.attn_norm_c = AdaLayerNormZero_Final(dim) if context_pre_only else AdaLayerNormZero(dim)+ self.attn_norm_x = AdaLayerNormZero(dim)+ self.attn = Attention(+ processor=JointAttnProcessor(),+ dim=dim,+ heads=heads,+ dim_head=dim_head,+ dropout=dropout,+ context_dim=dim,+ context_pre_only=context_pre_only,+ )++ if not context_pre_only:+ self.ff_norm_c = nn.LayerNorm(dim, elementwise_affine=False, eps=1e-6)+ self.ff_c = FeedForward(dim=dim, mult=ff_mult, dropout=dropout, approximate="tanh")+ else:+ self.ff_norm_c = None+ self.ff_c = None+ self.ff_norm_x = nn.LayerNorm(dim, elementwise_affine=False, eps=1e-6)+ self.ff_x = FeedForward(dim=dim, mult=ff_mult, dropout=dropout, approximate="tanh")++ def forward(self, x, c, t, mask=None, rope=None, c_rope=None): # x: noised input, c: context, t: time embedding+ # pre-norm & modulation for attention input+ if self.context_pre_only:+ norm_c = self.attn_norm_c(c, t)+ else:+ norm_c, c_gate_msa, c_shift_mlp, c_scale_mlp, c_gate_mlp = self.attn_norm_c(c, emb=t)+ norm_x, x_gate_msa, x_shift_mlp, x_scale_mlp, x_gate_mlp = self.attn_norm_x(x, emb=t)++ # attention+ x_attn_output, c_attn_output = self.attn(x=norm_x, c=norm_c, mask=mask, rope=rope, c_rope=c_rope)++ # process attention output for context c+ if self.context_pre_only:+ c = None+ else: # if not last layer+ c = c + c_gate_msa.unsqueeze(1) * c_attn_output++ norm_c = self.ff_norm_c(c) * (1 + c_scale_mlp[:, None]) + c_shift_mlp[:, None]+ c_ff_output = self.ff_c(norm_c)+ c = c + c_gate_mlp.unsqueeze(1) * c_ff_output++ # process attention output for input x+ x = x + x_gate_msa.unsqueeze(1) * x_attn_output++ norm_x = self.ff_norm_x(x) * (1 + x_scale_mlp[:, None]) + x_shift_mlp[:, None]+ x_ff_output = self.ff_x(norm_x)+ x = x + x_gate_mlp.unsqueeze(1) * x_ff_output++ return c, x+++# time step conditioning embedding+++class TimestepEmbedding(nn.Module):+ def __init__(self, dim, freq_embed_dim=256):+ super().__init__()+ self.time_embed = SinusPositionEmbedding(freq_embed_dim)+ self.time_mlp = nn.Sequential(nn.Linear(freq_embed_dim, dim), nn.SiLU(), nn.Linear(dim, dim))++ def forward(self, timestep: float["b"]): # noqa: F821+ time_hidden = self.time_embed(timestep)+ time_hidden = time_hidden.to(timestep.dtype)+ time = self.time_mlp(time_hidden) # b d+ return timediff --git a/freevideo_engine/dubbing/cosyvoice3/flow/__init__.py b/freevideo_engine/dubbing/cosyvoice3/flow/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/freevideo_engine/dubbing/cosyvoice3/flow/flow.py b/freevideo_engine/dubbing/cosyvoice3/flow/flow.pynew file mode 100644index 0000000..02f92f6--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/flow/flow.py@@ -0,0 +1,103 @@+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Zhihao Du)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# Modified by FreeVideo: reduced to the CosyVoice3 DiT flow inference path.+from typing import Dict++import torch+import torch.nn as nn+from torch.nn import functional as F+from omegaconf import DictConfig++from freevideo_engine.dubbing.cosyvoice3.utils.mask import make_pad_mask+++class CausalMaskedDiffWithDiT(torch.nn.Module):+ def __init__(self,+ input_size: int = 512,+ output_size: int = 80,+ spk_embed_dim: int = 192,+ output_type: str = "mel",+ vocab_size: int = 4096,+ input_frame_rate: int = 50,+ only_mask_loss: bool = True,+ token_mel_ratio: int = 2,+ pre_lookahead_len: int = 3,+ pre_lookahead_layer: torch.nn.Module = None,+ decoder: torch.nn.Module = None,+ decoder_conf: Dict = {'in_channels': 240, 'out_channel': 80, 'spk_emb_dim': 80, 'n_spks': 1,+ 'cfm_params': DictConfig({'sigma_min': 1e-06, 'solver': 'euler', 't_scheduler': 'cosine',+ 'training_cfg_rate': 0.2, 'inference_cfg_rate': 0.7, 'reg_loss_type': 'l1'}),+ 'decoder_params': {'channels': [256, 256], 'dropout': 0.0, 'attention_head_dim': 64,+ 'n_blocks': 4, 'num_mid_blocks': 12, 'num_heads': 8, 'act_fn': 'gelu'}}):+ super().__init__()+ self.input_size = input_size+ self.output_size = output_size+ self.decoder_conf = decoder_conf+ self.vocab_size = vocab_size+ self.output_type = output_type+ self.input_frame_rate = input_frame_rate+ self.input_embedding = nn.Embedding(vocab_size, input_size)+ self.spk_embed_affine_layer = torch.nn.Linear(spk_embed_dim, output_size)+ self.pre_lookahead_len = pre_lookahead_len+ self.pre_lookahead_layer = pre_lookahead_layer+ self.decoder = decoder+ self.only_mask_loss = only_mask_loss+ self.token_mel_ratio = token_mel_ratio++ @torch.inference_mode()+ def inference(self,+ token,+ token_len,+ prompt_token,+ prompt_token_len,+ prompt_feat,+ prompt_feat_len,+ embedding,+ streaming,+ finalize):+ assert token.shape[0] == 1+ # xvec projection+ embedding = F.normalize(embedding, dim=1)+ embedding = self.spk_embed_affine_layer(embedding)++ # concat text and prompt_text+ token, token_len = torch.concat([prompt_token, token], dim=1), prompt_token_len + token_len+ mask = (~make_pad_mask(token_len)).unsqueeze(-1).to(embedding)+ token = self.input_embedding(torch.clamp(token, min=0)) * mask++ # text encode+ if finalize is True:+ h = self.pre_lookahead_layer(token)+ else:+ h = self.pre_lookahead_layer(token[:, :-self.pre_lookahead_len], context=token[:, -self.pre_lookahead_len:])+ h = h.repeat_interleave(self.token_mel_ratio, dim=1)+ mel_len1, mel_len2 = prompt_feat.shape[1], h.shape[1] - prompt_feat.shape[1]++ # get conditions+ conds = torch.zeros([1, mel_len1 + mel_len2, self.output_size], device=token.device).to(h.dtype)+ conds[:, :mel_len1] = prompt_feat+ conds = conds.transpose(1, 2)++ mask = (~make_pad_mask(torch.tensor([mel_len1 + mel_len2]))).to(h)+ feat, _ = self.decoder(+ mu=h.transpose(1, 2).contiguous(),+ mask=mask.unsqueeze(1),+ spks=embedding,+ cond=conds,+ n_timesteps=10,+ streaming=streaming+ )+ feat = feat[:, :, mel_len1:]+ assert feat.shape[2] == mel_len2+ return feat.float(), Nonediff --git a/freevideo_engine/dubbing/cosyvoice3/flow/flow_matching.py b/freevideo_engine/dubbing/cosyvoice3/flow/flow_matching.pynew file mode 100644index 0000000..eab641d--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/flow/flow_matching.py@@ -0,0 +1,238 @@+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Zhihao Du)+# 2025 Alibaba Inc (authors: Xiang Lyu, Bofan Zhou)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+import torch+import torch.nn.functional as F+from freevideo_engine.dubbing.cosyvoice3.utils.common import set_all_random_seed+++class BASECFM(torch.nn.Module):+ # inlined from Matcha-TTS (MIT): only the attributes the subclass uses+ def __init__(self, n_feats, cfm_params, n_spks=1, spk_emb_dim=128):+ super().__init__()+ self.n_feats = n_feats+ self.n_spks = n_spks+ self.spk_emb_dim = spk_emb_dim+ self.solver = cfm_params.solver+ self.sigma_min = getattr(cfm_params, 'sigma_min', 1e-4)+ self.estimator = None+++class ConditionalCFM(BASECFM):+ def __init__(self, in_channels, cfm_params, n_spks=1, spk_emb_dim=64, estimator: torch.nn.Module = None):+ super().__init__(+ n_feats=in_channels,+ cfm_params=cfm_params,+ n_spks=n_spks,+ spk_emb_dim=spk_emb_dim,+ )+ self.t_scheduler = cfm_params.t_scheduler+ self.training_cfg_rate = cfm_params.training_cfg_rate+ self.inference_cfg_rate = cfm_params.inference_cfg_rate+ in_channels = in_channels + (spk_emb_dim if n_spks > 0 else 0)+ # Just change the architecture of the estimator here+ self.estimator = estimator++ @torch.inference_mode()+ def forward(self, mu, mask, n_timesteps, temperature=1.0, spks=None, cond=None, prompt_len=0, cache=torch.zeros(1, 80, 0, 2)):+ """Forward diffusion++ Args:+ mu (torch.Tensor): output of encoder+ shape: (batch_size, n_feats, mel_timesteps)+ mask (torch.Tensor): output_mask+ shape: (batch_size, 1, mel_timesteps)+ n_timesteps (int): number of diffusion steps+ temperature (float, optional): temperature for scaling noise. Defaults to 1.0.+ spks (torch.Tensor, optional): speaker ids. Defaults to None.+ shape: (batch_size, spk_emb_dim)+ cond: Not used but kept for future purposes++ Returns:+ sample: generated mel-spectrogram+ shape: (batch_size, n_feats, mel_timesteps)+ """++ z = torch.randn_like(mu).to(mu.device).to(mu.dtype) * temperature+ cache_size = cache.shape[2]+ # fix prompt and overlap part mu and z+ if cache_size != 0:+ z[:, :, :cache_size] = cache[:, :, :, 0]+ mu[:, :, :cache_size] = cache[:, :, :, 1]+ z_cache = torch.concat([z[:, :, :prompt_len], z[:, :, -34:]], dim=2)+ mu_cache = torch.concat([mu[:, :, :prompt_len], mu[:, :, -34:]], dim=2)+ cache = torch.stack([z_cache, mu_cache], dim=-1)++ t_span = torch.linspace(0, 1, n_timesteps + 1, device=mu.device, dtype=mu.dtype)+ if self.t_scheduler == 'cosine':+ t_span = 1 - torch.cos(t_span * 0.5 * torch.pi)+ return self.solve_euler(z, t_span=t_span, mu=mu, mask=mask, spks=spks, cond=cond), cache++ def solve_euler(self, x, t_span, mu, mask, spks, cond, streaming=False):+ """+ Fixed euler solver for ODEs.+ Args:+ x (torch.Tensor): random noise+ t_span (torch.Tensor): n_timesteps interpolated+ shape: (n_timesteps + 1,)+ mu (torch.Tensor): output of encoder+ shape: (batch_size, n_feats, mel_timesteps)+ mask (torch.Tensor): output_mask+ shape: (batch_size, 1, mel_timesteps)+ spks (torch.Tensor, optional): speaker ids. Defaults to None.+ shape: (batch_size, spk_emb_dim)+ cond: Not used but kept for future purposes+ """+ t, _, dt = t_span[0], t_span[-1], t_span[1] - t_span[0]+ t = t.unsqueeze(dim=0)++ # I am storing this because I can later plot it by putting a debugger here and saving it to a file+ # Or in future might add like a return_all_steps flag+ sol = []++ # Do not use concat, it may cause memory format changed and trt infer with wrong results!+ # NOTE when flow run in amp mode, x.dtype is float32, which cause nan in trt fp16 inference, so set dtype=spks.dtype+ x_in = torch.zeros([2, 80, x.size(2)], device=x.device, dtype=spks.dtype)+ mask_in = torch.zeros([2, 1, x.size(2)], device=x.device, dtype=spks.dtype)+ mu_in = torch.zeros([2, 80, x.size(2)], device=x.device, dtype=spks.dtype)+ t_in = torch.zeros([2], device=x.device, dtype=spks.dtype)+ spks_in = torch.zeros([2, 80], device=x.device, dtype=spks.dtype)+ cond_in = torch.zeros([2, 80, x.size(2)], device=x.device, dtype=spks.dtype)+ for step in range(1, len(t_span)):+ # Classifier-Free Guidance inference introduced in VoiceBox+ x_in[:] = x+ mask_in[:] = mask+ mu_in[0] = mu+ t_in[:] = t.unsqueeze(0)+ spks_in[0] = spks+ cond_in[0] = cond+ dphi_dt = self.forward_estimator(+ x_in, mask_in,+ mu_in, t_in,+ spks_in,+ cond_in,+ streaming+ )+ dphi_dt, cfg_dphi_dt = torch.split(dphi_dt, [x.size(0), x.size(0)], dim=0)+ dphi_dt = ((1.0 + self.inference_cfg_rate) * dphi_dt - self.inference_cfg_rate * cfg_dphi_dt)+ x = x + dt * dphi_dt+ t = t + dt+ sol.append(x)+ if step < len(t_span) - 1:+ dt = t_span[step + 1] - t++ return sol[-1].float()++ def forward_estimator(self, x, mask, mu, t, spks, cond, streaming=False):+ if isinstance(self.estimator, torch.nn.Module):+ return self.estimator(x, mask, mu, t, spks, cond, streaming=streaming)+ else:+ [estimator, stream], trt_engine = self.estimator.acquire_estimator()+ # NOTE need to synchronize when switching stream+ torch.cuda.current_stream().synchronize()+ with stream:+ estimator.set_input_shape('x', (2, 80, x.size(2)))+ estimator.set_input_shape('mask', (2, 1, x.size(2)))+ estimator.set_input_shape('mu', (2, 80, x.size(2)))+ estimator.set_input_shape('t', (2,))+ estimator.set_input_shape('spks', (2, 80))+ estimator.set_input_shape('cond', (2, 80, x.size(2)))+ data_ptrs = [x.contiguous().data_ptr(),+ mask.contiguous().data_ptr(),+ mu.contiguous().data_ptr(),+ t.contiguous().data_ptr(),+ spks.contiguous().data_ptr(),+ cond.contiguous().data_ptr(),+ x.data_ptr()]+ for i, j in enumerate(data_ptrs):+ estimator.set_tensor_address(trt_engine.get_tensor_name(i), j)+ # run trt engine+ assert estimator.execute_async_v3(torch.cuda.current_stream().cuda_stream) is True+ torch.cuda.current_stream().synchronize()+ self.estimator.release_estimator(estimator, stream)+ return x++ def compute_loss(self, x1, mask, mu, spks=None, cond=None, streaming=False):+ """Computes diffusion loss++ Args:+ x1 (torch.Tensor): Target+ shape: (batch_size, n_feats, mel_timesteps)+ mask (torch.Tensor): target mask+ shape: (batch_size, 1, mel_timesteps)+ mu (torch.Tensor): output of encoder+ shape: (batch_size, n_feats, mel_timesteps)+ spks (torch.Tensor, optional): speaker embedding. Defaults to None.+ shape: (batch_size, spk_emb_dim)++ Returns:+ loss: conditional flow matching loss+ y: conditional flow+ shape: (batch_size, n_feats, mel_timesteps)+ """+ b, _, t = mu.shape++ # random timestep+ t = torch.rand([b, 1, 1], device=mu.device, dtype=mu.dtype)++ # sample noise p(x_0)+ z = torch.randn_like(x1)++ y = (1 - (1 - self.sigma_min) * t) * z + t * x1+ u = x1 - (1 - self.sigma_min) * z++ # during training, we randomly drop condition to trade off mode coverage and sample fidelity+ if self.training_cfg_rate > 0:+ cfg_mask = torch.rand(b, device=x1.device) > self.training_cfg_rate+ mu = mu * cfg_mask.view(-1, 1, 1)+ spks = spks * cfg_mask.view(-1, 1)+ cond = cond * cfg_mask.view(-1, 1, 1)++ pred = self.estimator(y, mask, mu, t.squeeze(), spks, cond, streaming=streaming)+ loss = F.mse_loss(pred * mask, u * mask, reduction="sum") / (torch.sum(mask) * u.shape[1])+ return loss, y+++class CausalConditionalCFM(ConditionalCFM):+ def __init__(self, in_channels, cfm_params, n_spks=1, spk_emb_dim=64, estimator: torch.nn.Module = None):+ super().__init__(in_channels, cfm_params, n_spks, spk_emb_dim, estimator)+ set_all_random_seed(0)+ self.rand_noise = torch.randn([1, 80, 50 * 300])++ @torch.inference_mode()+ def forward(self, mu, mask, n_timesteps, temperature=1.0, spks=None, cond=None, streaming=False):+ """Forward diffusion++ Args:+ mu (torch.Tensor): output of encoder+ shape: (batch_size, n_feats, mel_timesteps)+ mask (torch.Tensor): output_mask+ shape: (batch_size, 1, mel_timesteps)+ n_timesteps (int): number of diffusion steps+ temperature (float, optional): temperature for scaling noise. Defaults to 1.0.+ spks (torch.Tensor, optional): speaker ids. Defaults to None.+ shape: (batch_size, spk_emb_dim)+ cond: Not used but kept for future purposes++ Returns:+ sample: generated mel-spectrogram+ shape: (batch_size, n_feats, mel_timesteps)+ """++ z = self.rand_noise[:, :, :mu.size(2)].to(mu.device).to(mu.dtype) * temperature+ # fix prompt and overlap part mu and z+ t_span = torch.linspace(0, 1, n_timesteps + 1, device=mu.device, dtype=mu.dtype)+ if self.t_scheduler == 'cosine':+ t_span = 1 - torch.cos(t_span * 0.5 * torch.pi)+ return self.solve_euler(z, t_span=t_span, mu=mu, mask=mask, spks=spks, cond=cond, streaming=streaming), Nonediff --git a/freevideo_engine/dubbing/cosyvoice3/hifigan/__init__.py b/freevideo_engine/dubbing/cosyvoice3/hifigan/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/freevideo_engine/dubbing/cosyvoice3/hifigan/f0_predictor.py b/freevideo_engine/dubbing/cosyvoice3/hifigan/f0_predictor.pynew file mode 100644index 0000000..25abfde--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/hifigan/f0_predictor.py@@ -0,0 +1,103 @@+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Kai Hu)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+import torch+import torch.nn as nn+try:+ from torch.nn.utils.parametrizations import weight_norm+except ImportError:+ from torch.nn.utils import weight_norm+from freevideo_engine.dubbing.cosyvoice3.transformer.convolution import CausalConv1d+++class ConvRNNF0Predictor(nn.Module):+ def __init__(self,+ num_class: int = 1,+ in_channels: int = 80,+ cond_channels: int = 512+ ):+ super().__init__()++ self.num_class = num_class+ self.condnet = nn.Sequential(+ weight_norm(+ nn.Conv1d(in_channels, cond_channels, kernel_size=3, padding=1)+ ),+ nn.ELU(),+ weight_norm(+ nn.Conv1d(cond_channels, cond_channels, kernel_size=3, padding=1)+ ),+ nn.ELU(),+ weight_norm(+ nn.Conv1d(cond_channels, cond_channels, kernel_size=3, padding=1)+ ),+ nn.ELU(),+ weight_norm(+ nn.Conv1d(cond_channels, cond_channels, kernel_size=3, padding=1)+ ),+ nn.ELU(),+ weight_norm(+ nn.Conv1d(cond_channels, cond_channels, kernel_size=3, padding=1)+ ),+ nn.ELU(),+ )+ self.classifier = nn.Linear(in_features=cond_channels, out_features=self.num_class)++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ x = self.condnet(x)+ x = x.transpose(1, 2)+ return torch.abs(self.classifier(x).squeeze(-1))+++class CausalConvRNNF0Predictor(nn.Module):+ def __init__(self,+ num_class: int = 1,+ in_channels: int = 80,+ cond_channels: int = 512+ ):+ super().__init__()++ self.num_class = num_class+ self.condnet = nn.Sequential(+ weight_norm(+ CausalConv1d(in_channels, cond_channels, kernel_size=4, causal_type='right')+ ),+ nn.ELU(),+ weight_norm(+ CausalConv1d(cond_channels, cond_channels, kernel_size=3, causal_type='left')+ ),+ nn.ELU(),+ weight_norm(+ CausalConv1d(cond_channels, cond_channels, kernel_size=3, causal_type='left')+ ),+ nn.ELU(),+ weight_norm(+ CausalConv1d(cond_channels, cond_channels, kernel_size=3, causal_type='left')+ ),+ nn.ELU(),+ weight_norm(+ CausalConv1d(cond_channels, cond_channels, kernel_size=3, causal_type='left')+ ),+ nn.ELU(),+ )+ self.classifier = nn.Linear(in_features=cond_channels, out_features=self.num_class)++ def forward(self, x: torch.Tensor, finalize: bool = True) -> torch.Tensor:+ if finalize is True:+ x = self.condnet[0](x)+ else:+ x = self.condnet[0](x[:, :, :-self.condnet[0].causal_padding], x[:, :, -self.condnet[0].causal_padding:])+ for i in range(1, len(self.condnet)):+ x = self.condnet[i](x)+ x = x.transpose(1, 2)+ return torch.abs(self.classifier(x).squeeze(-1))diff --git a/freevideo_engine/dubbing/cosyvoice3/hifigan/generator.py b/freevideo_engine/dubbing/cosyvoice3/hifigan/generator.pynew file mode 100644index 0000000..3766c84--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/hifigan/generator.py@@ -0,0 +1,726 @@+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Kai Hu)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""HIFI-GAN"""++from typing import Dict, Optional, List+import numpy as np+from scipy.signal import get_window+import torch+import torch.nn as nn+import torch.nn.functional as F+from torch.nn import Conv1d+from torch.nn import ConvTranspose1d+from torch.nn.utils import remove_weight_norm+try:+ from torch.nn.utils.parametrizations import weight_norm+except ImportError:+ from torch.nn.utils import weight_norm+from torch.distributions.uniform import Uniform+from freevideo_engine.dubbing.cosyvoice3.transformer.convolution import CausalConv1d, CausalConv1dDownSample, CausalConv1dUpsample+from freevideo_engine.dubbing.cosyvoice3.transformer.activation import Snake+from freevideo_engine.dubbing.cosyvoice3.utils.common import get_padding+from freevideo_engine.dubbing.cosyvoice3.utils.common import init_weights+++"""hifigan based generator implementation.++This code is modified from https://github.com/jik876/hifi-gan+ ,https://github.com/kan-bayashi/ParallelWaveGAN and+ https://github.com/NVIDIA/BigVGAN++"""+++class ResBlock(torch.nn.Module):+ """Residual block module in HiFiGAN/BigVGAN."""+ def __init__(+ self,+ channels: int = 512,+ kernel_size: int = 3,+ dilations: List[int] = [1, 3, 5],+ causal: bool = False,+ ):+ super(ResBlock, self).__init__()+ self.causal = causal+ self.convs1 = nn.ModuleList()+ self.convs2 = nn.ModuleList()++ for dilation in dilations:+ self.convs1.append(+ weight_norm(+ Conv1d(+ channels,+ channels,+ kernel_size,+ 1,+ dilation=dilation,+ padding=get_padding(kernel_size, dilation)) if causal is False else+ CausalConv1d(+ channels,+ channels,+ kernel_size,+ 1,+ dilation=dilation,+ causal_type='left'+ )+ )+ )+ self.convs2.append(+ weight_norm(+ Conv1d(+ channels,+ channels,+ kernel_size,+ 1,+ dilation=1,+ padding=get_padding(kernel_size, 1)) if causal is False else+ CausalConv1d(+ channels,+ channels,+ kernel_size,+ 1,+ dilation=1,+ causal_type='left'+ )+ )+ )+ self.convs1.apply(init_weights)+ self.convs2.apply(init_weights)+ self.activations1 = nn.ModuleList([+ Snake(channels, alpha_logscale=False)+ for _ in range(len(self.convs1))+ ])+ self.activations2 = nn.ModuleList([+ Snake(channels, alpha_logscale=False)+ for _ in range(len(self.convs2))+ ])++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ for idx in range(len(self.convs1)):+ xt = self.activations1[idx](x)+ xt = self.convs1[idx](xt)+ xt = self.activations2[idx](xt)+ xt = self.convs2[idx](xt)+ x = xt + x+ return x++ def remove_weight_norm(self):+ for idx in range(len(self.convs1)):+ remove_weight_norm(self.convs1[idx])+ remove_weight_norm(self.convs2[idx])+++class SineGen(torch.nn.Module):+ """ Definition of sine generator+ SineGen(samp_rate, harmonic_num = 0,+ sine_amp = 0.1, noise_std = 0.003,+ voiced_threshold = 0,+ flag_for_pulse=False)+ samp_rate: sampling rate in Hz+ harmonic_num: number of harmonic overtones (default 0)+ sine_amp: amplitude of sine-wavefrom (default 0.1)+ noise_std: std of Gaussian noise (default 0.003)+ voiced_thoreshold: F0 threshold for U/V classification (default 0)+ flag_for_pulse: this SinGen is used inside PulseGen (default False)+ Note: when flag_for_pulse is True, the first time step of a voiced+ segment is always sin(np.pi) or cos(0)+ """++ def __init__(self, samp_rate, harmonic_num=0,+ sine_amp=0.1, noise_std=0.003,+ voiced_threshold=0):+ super(SineGen, self).__init__()+ self.sine_amp = sine_amp+ self.noise_std = noise_std+ self.harmonic_num = harmonic_num+ self.sampling_rate = samp_rate+ self.voiced_threshold = voiced_threshold++ def _f02uv(self, f0):+ # generate uv signal+ uv = (f0 > self.voiced_threshold).type(torch.float32)+ return uv++ @torch.no_grad()+ def forward(self, f0):+ """ sine_tensor, uv = forward(f0)+ input F0: tensor(batchsize=1, dim=1, length)+ f0 for unvoiced steps should be 0+ output sine_tensor: tensor(batchsize=1, length, dim)+ output uv: tensor(batchsize=1, length, 1)+ """+ f0 = f0.transpose(1, 2)+ F_mat = torch.zeros((f0.size(0), self.harmonic_num + 1, f0.size(-1))).to(f0.device)+ for i in range(self.harmonic_num + 1):+ F_mat[:, i: i + 1, :] = f0 * (i + 1) / self.sampling_rate++ theta_mat = 2 * np.pi * (torch.cumsum(F_mat, dim=-1) % 1)+ u_dist = Uniform(low=-np.pi, high=np.pi)+ phase_vec = u_dist.sample(sample_shape=(f0.size(0), self.harmonic_num + 1, 1)).to(F_mat.device)+ phase_vec[:, 0, :] = 0++ # generate sine waveforms+ sine_waves = self.sine_amp * torch.sin(theta_mat + phase_vec)++ # generate uv signal+ uv = self._f02uv(f0)++ # noise: for unvoiced should be similar to sine_amp+ # std = self.sine_amp/3 -> max value ~ self.sine_amp+ # . for voiced regions is self.noise_std+ noise_amp = uv * self.noise_std + (1 - uv) * self.sine_amp / 3+ noise = noise_amp * torch.randn_like(sine_waves)++ # first: set the unvoiced part to 0 by uv+ # then: additive noise+ sine_waves = sine_waves * uv + noise+ return sine_waves.transpose(1, 2), uv.transpose(1, 2), noise+++class SineGen2(torch.nn.Module):+ """ Definition of sine generator+ SineGen(samp_rate, harmonic_num = 0,+ sine_amp = 0.1, noise_std = 0.003,+ voiced_threshold = 0,+ flag_for_pulse=False)+ samp_rate: sampling rate in Hz+ harmonic_num: number of harmonic overtones (default 0)+ sine_amp: amplitude of sine-wavefrom (default 0.1)+ noise_std: std of Gaussian noise (default 0.003)+ voiced_thoreshold: F0 threshold for U/V classification (default 0)+ flag_for_pulse: this SinGen is used inside PulseGen (default False)+ Note: when flag_for_pulse is True, the first time step of a voiced+ segment is always sin(np.pi) or cos(0)+ """++ def __init__(self, samp_rate, upsample_scale, harmonic_num=0,+ sine_amp=0.1, noise_std=0.003,+ voiced_threshold=0,+ flag_for_pulse=False,+ causal=False):+ super(SineGen2, self).__init__()+ self.sine_amp = sine_amp+ self.noise_std = noise_std+ self.harmonic_num = harmonic_num+ self.dim = self.harmonic_num + 1+ self.sampling_rate = samp_rate+ self.voiced_threshold = voiced_threshold+ self.flag_for_pulse = flag_for_pulse+ self.upsample_scale = upsample_scale+ self.causal = causal+ if causal is True:+ self.rand_ini = torch.rand(1, 9)+ self.rand_ini[:, 0] = 0+ self.sine_waves = torch.rand(1, 300 * 24000, 9)++ def _f02uv(self, f0):+ # generate uv signal+ uv = (f0 > self.voiced_threshold).type(torch.float32)+ return uv++ def _f02sine(self, f0_values):+ """ f0_values: (batchsize, length, dim)+ where dim indicates fundamental tone and overtones+ """+ # convert to F0 in rad. The interger part n can be ignored+ # because 2 * np.pi * n doesn't affect phase+ rad_values = (f0_values / self.sampling_rate) % 1++ # initial phase noise (no noise for fundamental component)+ if self.training is False and self.causal is True:+ rad_values[:, 0, :] = rad_values[:, 0, :] + self.rand_ini.to(rad_values.device)+ else:+ rand_ini = torch.rand(f0_values.shape[0], f0_values.shape[2], device=f0_values.device)+ rand_ini[:, 0] = 0+ rad_values[:, 0, :] = rad_values[:, 0, :] + rand_ini++ # instantanouse phase sine[t] = sin(2*pi \sum_i=1 ^{t} rad)+ if not self.flag_for_pulse:+ rad_values = torch.nn.functional.interpolate(rad_values.transpose(1, 2),+ scale_factor=1 / self.upsample_scale,+ mode="linear").transpose(1, 2)++ phase = torch.cumsum(rad_values, dim=1) * 2 * np.pi+ phase = torch.nn.functional.interpolate(phase.transpose(1, 2) * self.upsample_scale,+ scale_factor=self.upsample_scale, mode="nearest" if self.causal is True else 'linear').transpose(1, 2)+ sines = torch.sin(phase)+ else:+ # If necessary, make sure that the first time step of every+ # voiced segments is sin(pi) or cos(0)+ # This is used for pulse-train generation++ # identify the last time step in unvoiced segments+ uv = self._f02uv(f0_values)+ uv_1 = torch.roll(uv, shifts=-1, dims=1)+ uv_1[:, -1, :] = 1+ u_loc = (uv < 1) * (uv_1 > 0)++ # get the instantanouse phase+ tmp_cumsum = torch.cumsum(rad_values, dim=1)+ # different batch needs to be processed differently+ for idx in range(f0_values.shape[0]):+ temp_sum = tmp_cumsum[idx, u_loc[idx, :, 0], :]+ temp_sum[1:, :] = temp_sum[1:, :] - temp_sum[0:-1, :]+ # stores the accumulation of i.phase within+ # each voiced segments+ tmp_cumsum[idx, :, :] = 0+ tmp_cumsum[idx, u_loc[idx, :, 0], :] = temp_sum++ # rad_values - tmp_cumsum: remove the accumulation of i.phase+ # within the previous voiced segment.+ i_phase = torch.cumsum(rad_values - tmp_cumsum, dim=1)++ # get the sines+ sines = torch.cos(i_phase * 2 * np.pi)+ return sines++ def forward(self, f0):+ """ sine_tensor, uv = forward(f0)+ input F0: tensor(batchsize=1, length, dim=1)+ f0 for unvoiced steps should be 0+ output sine_tensor: tensor(batchsize=1, length, dim)+ output uv: tensor(batchsize=1, length, 1)+ """+ # fundamental component+ fn = torch.multiply(f0, torch.FloatTensor([[range(1, self.harmonic_num + 2)]]).to(f0.device))++ # generate sine waveforms+ sine_waves = self._f02sine(fn) * self.sine_amp++ # generate uv signal+ uv = self._f02uv(f0)++ # noise: for unvoiced should be similar to sine_amp+ # std = self.sine_amp/3 -> max value ~ self.sine_amp+ # . for voiced regions is self.noise_std+ noise_amp = uv * self.noise_std + (1 - uv) * self.sine_amp / 3+ if self.training is False and self.causal is True:+ noise = noise_amp * self.sine_waves[:, :sine_waves.shape[1]].to(sine_waves.device)+ else:+ noise = noise_amp * torch.randn_like(sine_waves)++ # first: set the unvoiced part to 0 by uv+ # then: additive noise+ sine_waves = sine_waves * uv + noise+ return sine_waves, uv, noise+++class SourceModuleHnNSF(torch.nn.Module):+ """ SourceModule for hn-nsf+ SourceModule(sampling_rate, harmonic_num=0, sine_amp=0.1,+ add_noise_std=0.003, voiced_threshod=0)+ sampling_rate: sampling_rate in Hz+ harmonic_num: number of harmonic above F0 (default: 0)+ sine_amp: amplitude of sine source signal (default: 0.1)+ add_noise_std: std of additive Gaussian noise (default: 0.003)+ note that amplitude of noise in unvoiced is decided+ by sine_amp+ voiced_threshold: threhold to set U/V given F0 (default: 0)+ Sine_source, noise_source = SourceModuleHnNSF(F0_sampled)+ F0_sampled (batchsize, length, 1)+ Sine_source (batchsize, length, 1)+ noise_source (batchsize, length 1)+ uv (batchsize, length, 1)+ """++ def __init__(self, sampling_rate, upsample_scale, harmonic_num=0, sine_amp=0.1,+ add_noise_std=0.003, voiced_threshod=0, sinegen_type='1', causal=False):+ super(SourceModuleHnNSF, self).__init__()++ self.sine_amp = sine_amp+ self.noise_std = add_noise_std++ # to produce sine waveforms+ if sinegen_type == '1':+ self.l_sin_gen = SineGen(sampling_rate, harmonic_num, sine_amp, add_noise_std, voiced_threshod)+ else:+ self.l_sin_gen = SineGen2(sampling_rate, upsample_scale, harmonic_num, sine_amp, add_noise_std, voiced_threshod, causal=causal)++ # to merge source harmonics into a single excitation+ self.l_linear = torch.nn.Linear(harmonic_num + 1, 1)+ self.l_tanh = torch.nn.Tanh()+ self.causal = causal+ if causal is True:+ self.uv = torch.rand(1, 300 * 24000, 1)++ def forward(self, x):+ """+ Sine_source, noise_source = SourceModuleHnNSF(F0_sampled)+ F0_sampled (batchsize, length, 1)+ Sine_source (batchsize, length, 1)+ noise_source (batchsize, length 1)+ """+ # source for harmonic branch+ with torch.no_grad():+ sine_wavs, uv, _ = self.l_sin_gen(x)+ sine_merge = self.l_tanh(self.l_linear(sine_wavs))++ # source for noise branch, in the same shape as uv+ if self.training is False and self.causal is True:+ noise = self.uv[:, :uv.shape[1]] * self.sine_amp / 3+ else:+ noise = torch.randn_like(uv) * self.sine_amp / 3+ return sine_merge, noise, uv+++class HiFTGenerator(nn.Module):+ """+ HiFTNet Generator: Neural Source Filter + ISTFTNet+ https://arxiv.org/abs/2309.09493+ """+ def __init__(+ self,+ in_channels: int = 80,+ base_channels: int = 512,+ nb_harmonics: int = 8,+ sampling_rate: int = 22050,+ nsf_alpha: float = 0.1,+ nsf_sigma: float = 0.003,+ nsf_voiced_threshold: float = 10,+ upsample_rates: List[int] = [8, 8],+ upsample_kernel_sizes: List[int] = [16, 16],+ istft_params: Dict[str, int] = {"n_fft": 16, "hop_len": 4},+ resblock_kernel_sizes: List[int] = [3, 7, 11],+ resblock_dilation_sizes: List[List[int]] = [[1, 3, 5], [1, 3, 5], [1, 3, 5]],+ source_resblock_kernel_sizes: List[int] = [7, 11],+ source_resblock_dilation_sizes: List[List[int]] = [[1, 3, 5], [1, 3, 5]],+ lrelu_slope: float = 0.1,+ audio_limit: float = 0.99,+ f0_predictor: torch.nn.Module = None,+ ):+ super(HiFTGenerator, self).__init__()++ self.out_channels = 1+ self.nb_harmonics = nb_harmonics+ self.sampling_rate = sampling_rate+ self.istft_params = istft_params+ self.lrelu_slope = lrelu_slope+ self.audio_limit = audio_limit++ self.num_kernels = len(resblock_kernel_sizes)+ self.num_upsamples = len(upsample_rates)+ # NOTE in CosyVoice2, we use the original SineGen implementation+ self.m_source = SourceModuleHnNSF(+ sampling_rate=sampling_rate,+ upsample_scale=np.prod(upsample_rates) * istft_params["hop_len"],+ harmonic_num=nb_harmonics,+ sine_amp=nsf_alpha,+ add_noise_std=nsf_sigma,+ voiced_threshod=nsf_voiced_threshold,+ sinegen_type='1' if self.sampling_rate == 22050 else '2',+ causal=False)+ self.f0_upsamp = torch.nn.Upsample(scale_factor=np.prod(upsample_rates) * istft_params["hop_len"])++ self.conv_pre = weight_norm(+ Conv1d(in_channels, base_channels, 7, 1, padding=3)+ )++ # Up+ self.ups = nn.ModuleList()+ for i, (u, k) in enumerate(zip(upsample_rates, upsample_kernel_sizes)):+ self.ups.append(+ weight_norm(+ ConvTranspose1d(+ base_channels // (2**i),+ base_channels // (2**(i + 1)),+ k,+ u,+ padding=(k - u) // 2,+ )+ )+ )++ # Down+ self.source_downs = nn.ModuleList()+ self.source_resblocks = nn.ModuleList()+ downsample_rates = [1] + upsample_rates[::-1][:-1]+ downsample_cum_rates = np.cumprod(downsample_rates)+ for i, (u, k, d) in enumerate(zip(downsample_cum_rates[::-1], source_resblock_kernel_sizes, source_resblock_dilation_sizes)):+ if u == 1:+ self.source_downs.append(+ Conv1d(istft_params["n_fft"] + 2, base_channels // (2 ** (i + 1)), 1, 1)+ )+ else:+ self.source_downs.append(+ Conv1d(istft_params["n_fft"] + 2, base_channels // (2 ** (i + 1)), u * 2, u, padding=(u // 2))+ )++ self.source_resblocks.append(+ ResBlock(base_channels // (2 ** (i + 1)), k, d)+ )++ self.resblocks = nn.ModuleList()+ for i in range(len(self.ups)):+ ch = base_channels // (2**(i + 1))+ for _, (k, d) in enumerate(zip(resblock_kernel_sizes, resblock_dilation_sizes)):+ self.resblocks.append(ResBlock(ch, k, d))++ self.conv_post = weight_norm(Conv1d(ch, istft_params["n_fft"] + 2, 7, 1, padding=3))+ self.ups.apply(init_weights)+ self.conv_post.apply(init_weights)+ self.reflection_pad = nn.ReflectionPad1d((1, 0))+ self.stft_window = torch.from_numpy(get_window("hann", istft_params["n_fft"], fftbins=True).astype(np.float32))+ self.f0_predictor = f0_predictor++ def remove_weight_norm(self):+ print('Removing weight norm...')+ for l in self.ups:+ remove_weight_norm(l)+ for l in self.resblocks:+ l.remove_weight_norm()+ remove_weight_norm(self.conv_pre)+ remove_weight_norm(self.conv_post)+ self.m_source.remove_weight_norm()+ for l in self.source_downs:+ remove_weight_norm(l)+ for l in self.source_resblocks:+ l.remove_weight_norm()++ def _stft(self, x):+ spec = torch.stft(+ x,+ self.istft_params["n_fft"], self.istft_params["hop_len"], self.istft_params["n_fft"], window=self.stft_window.to(x.device),+ return_complex=True)+ spec = torch.view_as_real(spec) # [B, F, TT, 2]+ return spec[..., 0], spec[..., 1]++ def _istft(self, magnitude, phase):+ magnitude = torch.clip(magnitude, max=1e2)+ real = magnitude * torch.cos(phase)+ img = magnitude * torch.sin(phase)+ inverse_transform = torch.istft(torch.complex(real, img), self.istft_params["n_fft"], self.istft_params["hop_len"],+ self.istft_params["n_fft"], window=self.stft_window.to(magnitude.device))+ return inverse_transform++ def decode(self, x: torch.Tensor, s: torch.Tensor = torch.zeros(1, 1, 0)) -> torch.Tensor:+ s_stft_real, s_stft_imag = self._stft(s.squeeze(1))+ s_stft = torch.cat([s_stft_real, s_stft_imag], dim=1)++ x = self.conv_pre(x)+ for i in range(self.num_upsamples):+ x = F.leaky_relu(x, self.lrelu_slope)+ x = self.ups[i](x)++ if i == self.num_upsamples - 1:+ x = self.reflection_pad(x)++ # fusion+ si = self.source_downs[i](s_stft)+ si = self.source_resblocks[i](si)+ x = x + si++ xs = None+ for j in range(self.num_kernels):+ if xs is None:+ xs = self.resblocks[i * self.num_kernels + j](x)+ else:+ xs += self.resblocks[i * self.num_kernels + j](x)+ x = xs / self.num_kernels++ x = F.leaky_relu(x)+ x = self.conv_post(x)+ magnitude = torch.exp(x[:, :self.istft_params["n_fft"] // 2 + 1, :])+ phase = torch.sin(x[:, self.istft_params["n_fft"] // 2 + 1:, :]) # actually, sin is redundancy++ x = self._istft(magnitude, phase)+ x = torch.clamp(x, -self.audio_limit, self.audio_limit)+ return x++ def forward(+ self,+ batch: dict,+ device: torch.device,+ ) -> Dict[str, Optional[torch.Tensor]]:+ speech_feat = batch['speech_feat'].transpose(1, 2).to(device)+ # mel->f0+ f0 = self.f0_predictor(speech_feat)+ # f0->source+ s = self.f0_upsamp(f0[:, None]).transpose(1, 2) # bs,n,t+ s, _, _ = self.m_source(s)+ s = s.transpose(1, 2)+ # mel+source->speech+ generated_speech = self.decode(x=speech_feat, s=s)+ return generated_speech, f0++ @torch.inference_mode()+ def inference(self, speech_feat: torch.Tensor, cache_source: torch.Tensor = torch.zeros(1, 1, 0)) -> torch.Tensor:+ # mel->f0+ f0 = self.f0_predictor(speech_feat)+ # f0->source+ s = self.f0_upsamp(f0[:, None]).transpose(1, 2) # bs,n,t+ s, _, _ = self.m_source(s)+ s = s.transpose(1, 2)+ # use cache_source to avoid glitch+ if cache_source.shape[2] != 0:+ s[:, :, :cache_source.shape[2]] = cache_source+ generated_speech = self.decode(x=speech_feat, s=s)+ return generated_speech, s+++class CausalHiFTGenerator(HiFTGenerator):+ """+ HiFTNet Generator: Neural Source Filter + ISTFTNet+ https://arxiv.org/abs/2309.09493+ """+ def __init__(+ self,+ in_channels: int = 80,+ base_channels: int = 512,+ nb_harmonics: int = 8,+ sampling_rate: int = 22050,+ nsf_alpha: float = 0.1,+ nsf_sigma: float = 0.003,+ nsf_voiced_threshold: float = 10,+ upsample_rates: List[int] = [8, 8],+ upsample_kernel_sizes: List[int] = [16, 16],+ istft_params: Dict[str, int] = {"n_fft": 16, "hop_len": 4},+ resblock_kernel_sizes: List[int] = [3, 7, 11],+ resblock_dilation_sizes: List[List[int]] = [[1, 3, 5], [1, 3, 5], [1, 3, 5]],+ source_resblock_kernel_sizes: List[int] = [7, 11],+ source_resblock_dilation_sizes: List[List[int]] = [[1, 3, 5], [1, 3, 5]],+ lrelu_slope: float = 0.1,+ audio_limit: float = 0.99,+ conv_pre_look_right: int = 4,+ f0_predictor: torch.nn.Module = None,+ ):+ torch.nn.Module.__init__(self)++ self.out_channels = 1+ self.nb_harmonics = nb_harmonics+ self.sampling_rate = sampling_rate+ self.istft_params = istft_params+ self.lrelu_slope = lrelu_slope+ self.audio_limit = audio_limit++ self.num_kernels = len(resblock_kernel_sizes)+ self.num_upsamples = len(upsample_rates)+ self.m_source = SourceModuleHnNSF(+ sampling_rate=sampling_rate,+ upsample_scale=np.prod(upsample_rates) * istft_params["hop_len"],+ harmonic_num=nb_harmonics,+ sine_amp=nsf_alpha,+ add_noise_std=nsf_sigma,+ voiced_threshod=nsf_voiced_threshold,+ sinegen_type='1' if self.sampling_rate == 22050 else '2',+ causal=True)+ self.upsample_rates = upsample_rates+ self.f0_upsamp = torch.nn.Upsample(scale_factor=np.prod(upsample_rates) * istft_params["hop_len"])++ self.conv_pre = weight_norm(+ CausalConv1d(in_channels, base_channels, conv_pre_look_right + 1, 1, causal_type='right')+ )++ # Up+ self.ups = nn.ModuleList()+ for i, (u, k) in enumerate(zip(upsample_rates, upsample_kernel_sizes)):+ self.ups.append(+ weight_norm(+ CausalConv1dUpsample(+ base_channels // (2**i),+ base_channels // (2**(i + 1)),+ k,+ u,+ )+ )+ )++ # Down+ self.source_downs = nn.ModuleList()+ self.source_resblocks = nn.ModuleList()+ downsample_rates = [1] + upsample_rates[::-1][:-1]+ downsample_cum_rates = np.cumprod(downsample_rates)+ for i, (u, k, d) in enumerate(zip(downsample_cum_rates[::-1], source_resblock_kernel_sizes, source_resblock_dilation_sizes)):+ if u == 1:+ self.source_downs.append(+ CausalConv1d(istft_params["n_fft"] + 2, base_channels // (2 ** (i + 1)), 1, 1, causal_type='left')+ )+ else:+ self.source_downs.append(+ CausalConv1dDownSample(istft_params["n_fft"] + 2, base_channels // (2 ** (i + 1)), u * 2, u)+ )++ self.source_resblocks.append(+ ResBlock(base_channels // (2 ** (i + 1)), k, d, causal=True)+ )++ self.resblocks = nn.ModuleList()+ for i in range(len(self.ups)):+ ch = base_channels // (2**(i + 1))+ for _, (k, d) in enumerate(zip(resblock_kernel_sizes, resblock_dilation_sizes)):+ self.resblocks.append(ResBlock(ch, k, d, causal=True))++ self.conv_post = weight_norm(CausalConv1d(ch, istft_params["n_fft"] + 2, 7, 1, causal_type='left'))+ self.ups.apply(init_weights)+ self.conv_post.apply(init_weights)+ self.reflection_pad = nn.ReflectionPad1d((1, 0))+ self.stft_window = torch.from_numpy(get_window("hann", istft_params["n_fft"], fftbins=True).astype(np.float32))+ self.conv_pre_look_right = conv_pre_look_right+ self.f0_predictor = f0_predictor++ def decode(self, x: torch.Tensor, s: torch.Tensor = torch.zeros(1, 1, 0), finalize: bool = True) -> torch.Tensor:+ s_stft_real, s_stft_imag = self._stft(s.squeeze(1))+ if finalize is True:+ x = self.conv_pre(x)+ else:+ x = self.conv_pre(x[:, :, :-self.conv_pre_look_right], x[:, :, -self.conv_pre_look_right:])+ s_stft_real = s_stft_real[:, :, :-int(np.prod(self.upsample_rates) * self.conv_pre_look_right)]+ s_stft_imag = s_stft_imag[:, :, :-int(np.prod(self.upsample_rates) * self.conv_pre_look_right)]+ s_stft = torch.cat([s_stft_real, s_stft_imag], dim=1)++ for i in range(self.num_upsamples):+ x = F.leaky_relu(x, self.lrelu_slope)+ x = self.ups[i](x)++ if i == self.num_upsamples - 1:+ x = self.reflection_pad(x)++ # fusion+ si = self.source_downs[i](s_stft)+ si = self.source_resblocks[i](si)+ x = x + si++ xs = None+ for j in range(self.num_kernels):+ if xs is None:+ xs = self.resblocks[i * self.num_kernels + j](x)+ else:+ xs += self.resblocks[i * self.num_kernels + j](x)+ x = xs / self.num_kernels++ x = F.leaky_relu(x)+ x = self.conv_post(x)+ magnitude = torch.exp(x[:, :self.istft_params["n_fft"] // 2 + 1, :])+ phase = torch.sin(x[:, self.istft_params["n_fft"] // 2 + 1:, :]) # actually, sin is redundancy++ x = self._istft(magnitude, phase)+ if finalize is False:+ x = x[:, :-int(np.prod(self.upsample_rates) * self.istft_params['hop_len'])]+ x = torch.clamp(x, -self.audio_limit, self.audio_limit)+ return x++ @torch.inference_mode()+ def inference(self, speech_feat: torch.Tensor, finalize: bool = True) -> torch.Tensor:+ # mel->f0 NOTE f0_predictor precision is crucial for causal inference, move self.f0_predictor to cpu if necessary+ self.f0_predictor.to(torch.float64)+ f0 = self.f0_predictor(speech_feat.to(torch.float64), finalize=finalize).to(speech_feat)+ # f0->source+ s = self.f0_upsamp(f0[:, None]).transpose(1, 2) # bs,n,t+ s, _, _ = self.m_source(s)+ s = s.transpose(1, 2)+ if finalize is True:+ generated_speech = self.decode(x=speech_feat, s=s, finalize=finalize)+ else:+ generated_speech = self.decode(x=speech_feat[:, :, :-self.f0_predictor.condnet[0].causal_padding], s=s, finalize=finalize)+ return generated_speech, sdiff --git a/freevideo_engine/dubbing/cosyvoice3/llm/__init__.py b/freevideo_engine/dubbing/cosyvoice3/llm/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/freevideo_engine/dubbing/cosyvoice3/llm/llm.py b/freevideo_engine/dubbing/cosyvoice3/llm/llm.pynew file mode 100644index 0000000..6874fbb--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/llm/llm.py@@ -0,0 +1,119 @@+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Zhihao Du)+# 2025 Alibaba Inc (authors: Xiang Lyu, Yabin Li, Qihua, Shengqiang Li)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# Modified by FreeVideo: reduced to the CosyVoice3 inference path (no training, streaming+# or vLLM code); the Qwen2 backbone is built from its config and filled from llm.pt.+from typing import Callable, Generator, List++import torch+from torch import nn+from transformers import AutoConfig, Qwen2ForCausalLM+++class Qwen2Encoder(torch.nn.Module):+ def __init__(self, config_dir):+ super().__init__()+ config = AutoConfig.from_pretrained(config_dir)+ config.dtype = torch.float32+ self.model = Qwen2ForCausalLM(config).to(torch.float32)++ def forward_one_step(self, xs, cache=None):+ # single sequence, no padding: the causal mask comes from the cache.+ # (a one-step mask of length 1 would be applied against the whole KV cache by transformers>=5)+ outs = self.model(+ inputs_embeds=xs,+ attention_mask=None,+ output_hidden_states=True,+ return_dict=True,+ use_cache=True,+ past_key_values=cache,+ )+ return outs.hidden_states[-1], outs.past_key_values+++class CosyVoice3LM(torch.nn.Module):+ def __init__(+ self,+ llm_input_size: int,+ llm_output_size: int,+ speech_token_size: int,+ llm: torch.nn.Module,+ sampling: Callable,+ mix_ratio: List[int] = [5, 15],+ ):+ super().__init__()+ self.llm_input_size = llm_input_size+ self.llm_output_size = llm_output_size+ self.speech_token_size = speech_token_size+ self.sos = speech_token_size + 0+ self.eos_token = speech_token_size + 1+ self.task_id = speech_token_size + 2+ self.fill_token = speech_token_size + 3++ self.llm = llm+ self.llm_decoder = nn.Linear(llm_output_size, speech_token_size + 200, bias=False)+ self.speech_embedding = torch.nn.Embedding(speech_token_size + 200, llm_input_size)+ self.sampling = sampling+ self.mix_ratio = mix_ratio+ self.stop_token_ids = [speech_token_size + i for i in range(200)]++ def sampling_ids(self, weighted_scores: torch.Tensor, decoded_tokens: List, sampling: int, ignore_eos: bool = True):+ if ignore_eos is True:+ weighted_scores[self.speech_token_size] = -float('inf')+ return self.sampling(weighted_scores, decoded_tokens, sampling)++ @torch.inference_mode()+ def inference(+ self,+ text: torch.Tensor,+ text_len: torch.Tensor,+ prompt_text: torch.Tensor,+ prompt_text_len: torch.Tensor,+ prompt_speech_token: torch.Tensor,+ prompt_speech_token_len: torch.Tensor,+ embedding: torch.Tensor,+ sampling: int = 25,+ max_token_text_ratio: float = 20,+ min_token_text_ratio: float = 2,+ uuid: str = '',+ ) -> Generator[int, None, None]:+ device = text.device+ text = torch.concat([prompt_text, text], dim=1)+ text_len += prompt_text_len+ text_emb = self.llm.model.model.embed_tokens(text)+ # 151646 is the <|endofprompt|> token+ assert 151646 in text, '<|endofprompt|> not detected in CosyVoice3 text or prompt_text, check your input!'++ sos_emb = self.speech_embedding.weight[self.sos].reshape(1, 1, -1)+ task_id_emb = self.speech_embedding.weight[self.task_id].reshape(1, 1, -1)+ if prompt_speech_token_len != 0:+ prompt_speech_token_emb = self.speech_embedding(prompt_speech_token)+ else:+ prompt_speech_token_emb = torch.zeros(1, 0, self.llm_input_size, dtype=text_emb.dtype).to(device)+ lm_input = torch.concat([sos_emb, text_emb, task_id_emb, prompt_speech_token_emb], dim=1)++ min_len = int((text_len - prompt_text_len) * min_token_text_ratio)+ max_len = int((text_len - prompt_text_len) * max_token_text_ratio)++ out_tokens = []+ cache = None+ for i in range(max_len):+ y_pred, cache = self.llm.forward_one_step(lm_input, cache=cache)+ logp = self.llm_decoder(y_pred[:, -1]).log_softmax(dim=-1)+ top_ids = self.sampling_ids(logp.squeeze(dim=0), out_tokens, sampling, ignore_eos=True if i < min_len else False)+ if top_ids in self.stop_token_ids:+ break+ yield top_ids+ out_tokens.append(top_ids)+ lm_input = self.speech_embedding.weight[top_ids].reshape(1, 1, -1)diff --git a/freevideo_engine/dubbing/cosyvoice3/tokenizer/__init__.py b/freevideo_engine/dubbing/cosyvoice3/tokenizer/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/freevideo_engine/dubbing/cosyvoice3/tokenizer/tokenizer.py b/freevideo_engine/dubbing/cosyvoice3/tokenizer/tokenizer.pynew file mode 100644index 0000000..3fc95a5--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/tokenizer/tokenizer.py@@ -0,0 +1,107 @@+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# Modified by FreeVideo: reduced to the Qwen-based CosyVoice tokenizers (no tiktoken).+from functools import lru_cache++import torch+from transformers import AutoTokenizer+++class CosyVoice2Tokenizer():+ def __init__(self, token_path, skip_special_tokens=True):+ super().__init__()+ # NOTE: non-chat model, all these special tokens keep randomly initialized.+ special_tokens = {+ 'eos_token': '<|endoftext|>',+ 'pad_token': '<|endoftext|>',+ 'additional_special_tokens': [+ '<|im_start|>', '<|im_end|>', '<|endofprompt|>',+ '[breath]', '<strong>', '</strong>', '[noise]',+ '[laughter]', '[cough]', '[clucking]', '[accent]',+ '[quick_breath]',+ "<laughter>", "</laughter>",+ "[hissing]", "[sigh]", "[vocalized-noise]",+ "[lipsmack]", "[mn]"+ ]+ }+ self.special_tokens = special_tokens+ self.tokenizer = AutoTokenizer.from_pretrained(token_path)+ self.tokenizer.add_special_tokens(special_tokens)+ self.skip_special_tokens = skip_special_tokens++ def encode(self, text, **kwargs):+ tokens = self.tokenizer([text], return_tensors="pt")+ tokens = tokens["input_ids"][0].cpu().tolist()+ return tokens++ def decode(self, tokens):+ tokens = torch.tensor(tokens, dtype=torch.int64)+ text = self.tokenizer.batch_decode([tokens], skip_special_tokens=self.skip_special_tokens)[0]+ return text+++class CosyVoice3Tokenizer(CosyVoice2Tokenizer):+ def __init__(self, token_path, skip_special_tokens=True):+ # NOTE: non-chat model, all these special tokens keep randomly initialized.+ special_tokens = {+ 'eos_token': '<|endoftext|>',+ 'pad_token': '<|endoftext|>',+ 'additional_special_tokens': [+ '<|im_start|>', '<|im_end|>', '<|endofprompt|>',+ '[breath]', '<strong>', '</strong>', '[noise]',+ '[laughter]', '[cough]', '[clucking]', '[accent]',+ '[quick_breath]',+ "<laughter>", "</laughter>",+ "[hissing]", "[sigh]", "[vocalized-noise]",+ "[lipsmack]", "[mn]", "<|endofsystem|>",+ "[AA]", "[AA0]", "[AA1]", "[AA2]", "[AE]", "[AE0]", "[AE1]", "[AE2]", "[AH]", "[AH0]", "[AH1]", "[AH2]",+ "[AO]", "[AO0]", "[AO1]", "[AO2]", "[AW]", "[AW0]", "[AW1]", "[AW2]", "[AY]", "[AY0]", "[AY1]", "[AY2]",+ "[B]", "[CH]", "[D]", "[DH]", "[EH]", "[EH0]", "[EH1]", "[EH2]", "[ER]", "[ER0]", "[ER1]", "[ER2]", "[EY]",+ "[EY0]", "[EY1]", "[EY2]", "[F]", "[G]", "[HH]", "[IH]", "[IH0]", "[IH1]", "[IH2]", "[IY]", "[IY0]", "[IY1]",+ "[IY2]", "[JH]", "[K]", "[L]", "[M]", "[N]", "[NG]", "[OW]", "[OW0]", "[OW1]", "[OW2]", "[OY]", "[OY0]",+ "[OY1]", "[OY2]", "[P]", "[R]", "[S]", "[SH]", "[T]", "[TH]", "[UH]", "[UH0]", "[UH1]", "[UH2]", "[UW]",+ "[UW0]", "[UW1]", "[UW2]", "[V]", "[W]", "[Y]", "[Z]", "[ZH]",+ "[a]", "[ai]", "[an]", "[ang]", "[ao]", "[b]", "[c]", "[ch]", "[d]", "[e]", "[ei]", "[en]", "[eng]", "[f]",+ "[g]", "[h]", "[i]", "[ian]", "[in]", "[ing]", "[iu]", "[ià]", "[iàn]", "[iàng]", "[iào]", "[iá]", "[ián]",+ "[iáng]", "[iáo]", "[iè]", "[ié]", "[iòng]", "[ióng]", "[iù]", "[iú]", "[iā]", "[iān]", "[iāng]", "[iāo]",+ "[iē]", "[iě]", "[iōng]", "[iū]", "[iǎ]", "[iǎn]", "[iǎng]", "[iǎo]", "[iǒng]", "[iǔ]", "[j]", "[k]", "[l]",+ "[m]", "[n]", "[o]", "[ong]", "[ou]", "[p]", "[q]", "[r]", "[s]", "[sh]", "[t]", "[u]", "[uang]", "[ue]",+ "[un]", "[uo]", "[uà]", "[uài]", "[uàn]", "[uàng]", "[uá]", "[uái]", "[uán]", "[uáng]", "[uè]", "[ué]", "[uì]",+ "[uí]", "[uò]", "[uó]", "[uā]", "[uāi]", "[uān]", "[uāng]", "[uē]", "[uě]", "[uī]", "[uō]", "[uǎ]", "[uǎi]",+ "[uǎn]", "[uǎng]", "[uǐ]", "[uǒ]", "[vè]", "[w]", "[x]", "[y]", "[z]", "[zh]", "[à]", "[ài]", "[àn]", "[àng]",+ "[ào]", "[á]", "[ái]", "[án]", "[áng]", "[áo]", "[è]", "[èi]", "[èn]", "[èng]", "[èr]", "[é]", "[éi]", "[én]",+ "[éng]", "[ér]", "[ì]", "[ìn]", "[ìng]", "[í]", "[ín]", "[íng]", "[ò]", "[òng]", "[òu]", "[ó]", "[óng]", "[óu]",+ "[ù]", "[ùn]", "[ú]", "[ún]", "[ā]", "[āi]", "[ān]", "[āng]", "[āo]", "[ē]", "[ēi]", "[ēn]", "[ēng]", "[ě]",+ "[ěi]", "[ěn]", "[ěng]", "[ěr]", "[ī]", "[īn]", "[īng]", "[ō]", "[ōng]", "[ōu]", "[ū]", "[ūn]", "[ǎ]", "[ǎi]",+ "[ǎn]", "[ǎng]", "[ǎo]", "[ǐ]", "[ǐn]", "[ǐng]", "[ǒ]", "[ǒng]", "[ǒu]", "[ǔ]", "[ǔn]", "[ǘ]", "[ǚ]", "[ǜ]"+ ]+ }+ self.special_tokens = special_tokens+ self.tokenizer = AutoTokenizer.from_pretrained(token_path)+ self.tokenizer.add_special_tokens(special_tokens)+ self.skip_special_tokens = skip_special_tokens+++@lru_cache(maxsize=None)+def get_qwen_tokenizer(+ token_path: str,+ skip_special_tokens: bool,+ version: str = 'cosyvoice2'+):+ if version == 'cosyvoice2':+ return CosyVoice2Tokenizer(token_path=token_path, skip_special_tokens=skip_special_tokens)+ elif version == 'cosyvoice3':+ return CosyVoice3Tokenizer(token_path=token_path, skip_special_tokens=skip_special_tokens)+ else:+ raise ValueErrordiff --git a/freevideo_engine/dubbing/cosyvoice3/transformer/__init__.py b/freevideo_engine/dubbing/cosyvoice3/transformer/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/freevideo_engine/dubbing/cosyvoice3/transformer/activation.py b/freevideo_engine/dubbing/cosyvoice3/transformer/activation.pynew file mode 100644index 0000000..8cea548--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/transformer/activation.py@@ -0,0 +1,84 @@+# Copyright (c) 2020 Johns Hopkins University (Shinji Watanabe)+# 2020 Northwestern Polytechnical University (Pengcheng Guo)+# 2020 Mobvoi Inc (Binbin Zhang)+# 2024 Alibaba Inc (Xiang Lyu)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+"""Swish() activation function for Conformer."""++import torch+from torch import nn, sin, pow+from torch.nn import Parameter+++class Swish(torch.nn.Module):+ """Construct an Swish object."""++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ """Return Swish activation function."""+ return x * torch.sigmoid(x)+++# Implementation adapted from https://github.com/EdwardDixon/snake under the MIT license.+# LICENSE is in incl_licenses directory.+class Snake(nn.Module):+ '''+ Implementation of a sine-based periodic activation function+ Shape:+ - Input: (B, C, T)+ - Output: (B, C, T), same shape as the input+ Parameters:+ - alpha - trainable parameter+ References:+ - This activation function is from this paper by Liu Ziyin, Tilman Hartwig, Masahito Ueda:+ https://arxiv.org/abs/2006.08195+ Examples:+ >>> a1 = snake(256)+ >>> x = torch.randn(256)+ >>> x = a1(x)+ '''+ def __init__(self, in_features, alpha=1.0, alpha_trainable=True, alpha_logscale=False):+ '''+ Initialization.+ INPUT:+ - in_features: shape of the input+ - alpha: trainable parameter+ alpha is initialized to 1 by default, higher values = higher-frequency.+ alpha will be trained along with the rest of your model.+ '''+ super(Snake, self).__init__()+ self.in_features = in_features++ # initialize alpha+ self.alpha_logscale = alpha_logscale+ if self.alpha_logscale: # log scale alphas initialized to zeros+ self.alpha = Parameter(torch.zeros(in_features) * alpha)+ else: # linear scale alphas initialized to ones+ self.alpha = Parameter(torch.ones(in_features) * alpha)++ self.alpha.requires_grad = alpha_trainable++ self.no_div_by_zero = 0.000000001++ def forward(self, x):+ '''+ Forward pass of the function.+ Applies the function to the input elementwise.+ Snake ∶= x + 1/a * sin^2 (xa)+ '''+ alpha = self.alpha.unsqueeze(0).unsqueeze(-1) # line up with x to [B, C, T]+ if self.alpha_logscale:+ alpha = torch.exp(alpha)+ x = x + (1.0 / (alpha + self.no_div_by_zero)) * pow(sin(x * alpha), 2)++ return xdiff --git a/freevideo_engine/dubbing/cosyvoice3/transformer/convolution.py b/freevideo_engine/dubbing/cosyvoice3/transformer/convolution.pynew file mode 100644index 0000000..edb32b2--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/transformer/convolution.py@@ -0,0 +1,258 @@+# Copyright (c) 2020 Mobvoi Inc. (authors: Binbin Zhang, Di Wu)+# 2024 Alibaba Inc (Xiang Lyu)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# Modified from ESPnet(https://github.com/espnet/espnet)+"""ConvolutionModule definition."""++from typing import Tuple++import torch+from torch import nn+import torch.nn.functional as F+++class ConvolutionModule(nn.Module):+ """ConvolutionModule in Conformer model."""++ def __init__(self,+ channels: int,+ kernel_size: int = 15,+ activation: nn.Module = nn.ReLU(),+ norm: str = "batch_norm",+ causal: bool = False,+ bias: bool = True):+ """Construct an ConvolutionModule object.+ Args:+ channels (int): The number of channels of conv layers.+ kernel_size (int): Kernel size of conv layers.+ causal (int): Whether use causal convolution or not+ """+ super().__init__()++ self.pointwise_conv1 = nn.Conv1d(+ channels,+ 2 * channels,+ kernel_size=1,+ stride=1,+ padding=0,+ bias=bias,+ )+ # self.lorder is used to distinguish if it's a causal convolution,+ # if self.lorder > 0: it's a causal convolution, the input will be+ # padded with self.lorder frames on the left in forward.+ # else: it's a symmetrical convolution+ if causal:+ padding = 0+ self.lorder = kernel_size - 1+ else:+ # kernel_size should be an odd number for none causal convolution+ assert (kernel_size - 1) % 2 == 0+ padding = (kernel_size - 1) // 2+ self.lorder = 0+ self.depthwise_conv = nn.Conv1d(+ channels,+ channels,+ kernel_size,+ stride=1,+ padding=padding,+ groups=channels,+ bias=bias,+ )++ assert norm in ['batch_norm', 'layer_norm']+ if norm == "batch_norm":+ self.use_layer_norm = False+ self.norm = nn.BatchNorm1d(channels)+ else:+ self.use_layer_norm = True+ self.norm = nn.LayerNorm(channels)++ self.pointwise_conv2 = nn.Conv1d(+ channels,+ channels,+ kernel_size=1,+ stride=1,+ padding=0,+ bias=bias,+ )+ self.activation = activation++ def forward(+ self,+ x: torch.Tensor,+ mask_pad: torch.Tensor = torch.ones((0, 0, 0), dtype=torch.bool),+ cache: torch.Tensor = torch.zeros((0, 0, 0)),+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Compute convolution module.+ Args:+ x (torch.Tensor): Input tensor (#batch, time, channels).+ mask_pad (torch.Tensor): used for batch padding (#batch, 1, time),+ (0, 0, 0) means fake mask.+ cache (torch.Tensor): left context cache, it is only+ used in causal convolution (#batch, channels, cache_t),+ (0, 0, 0) meas fake cache.+ Returns:+ torch.Tensor: Output tensor (#batch, time, channels).+ """+ # exchange the temporal dimension and the feature dimension+ x = x.transpose(1, 2) # (#batch, channels, time)++ # mask batch padding+ if mask_pad.size(2) > 0: # time > 0+ x.masked_fill_(~mask_pad, 0.0)++ if self.lorder > 0:+ if cache.size(2) == 0: # cache_t == 0+ x = nn.functional.pad(x, (self.lorder, 0), 'constant', 0.0)+ else:+ assert cache.size(0) == x.size(0) # equal batch+ assert cache.size(1) == x.size(1) # equal channel+ x = torch.cat((cache, x), dim=2)+ assert (x.size(2) > self.lorder)+ new_cache = x[:, :, -self.lorder:]+ else:+ # It's better we just return None if no cache is required,+ # However, for JIT export, here we just fake one tensor instead of+ # None.+ new_cache = torch.zeros((0, 0, 0), dtype=x.dtype, device=x.device)++ # GLU mechanism+ x = self.pointwise_conv1(x) # (batch, 2*channel, dim)+ x = nn.functional.glu(x, dim=1) # (batch, channel, dim)++ # 1D Depthwise Conv+ x = self.depthwise_conv(x)+ if self.use_layer_norm:+ x = x.transpose(1, 2)+ x = self.activation(self.norm(x))+ if self.use_layer_norm:+ x = x.transpose(1, 2)+ x = self.pointwise_conv2(x)+ # mask batch padding+ if mask_pad.size(2) > 0: # time > 0+ x.masked_fill_(~mask_pad, 0.0)++ return x.transpose(1, 2), new_cache+++# NOTE(Xiang Lyu) causal conv module used in convolution-based vocoder+class CausalConv1d(torch.nn.Conv1d):+ def __init__(+ self,+ in_channels: int,+ out_channels: int,+ kernel_size: int,+ stride: int = 1,+ dilation: int = 1,+ groups: int = 1,+ bias: bool = True,+ padding_mode: str = 'zeros',+ causal_type: str = 'left',+ device=None,+ dtype=None+ ) -> None:+ super(CausalConv1d, self).__init__(in_channels, out_channels,+ kernel_size, stride=1,+ padding=0, dilation=dilation,+ groups=groups, bias=bias,+ padding_mode=padding_mode,+ device=device, dtype=dtype)+ assert stride == 1+ self.causal_padding = int((kernel_size * dilation - dilation) / 2) * 2 + (kernel_size + 1) % 2+ assert causal_type in ['left', 'right']+ self.causal_type = causal_type++ def forward(self, x: torch.Tensor, cache: torch.Tensor = torch.zeros(0, 0, 0)) -> Tuple[torch.Tensor]:+ input_timestep = x.shape[2]+ if cache.size(2) == 0:+ cache = torch.zeros(x.shape[0], x.shape[1], self.causal_padding).to(x)+ assert cache.size(2) == self.causal_padding+ if self.causal_type == 'left':+ x = torch.concat([cache, x], dim=2)+ else:+ x = torch.concat([x, cache], dim=2)+ x = super(CausalConv1d, self).forward(x)+ assert x.shape[2] == input_timestep+ return x+++class CausalConv1dDownSample(torch.nn.Conv1d):+ def __init__(+ self,+ in_channels: int,+ out_channels: int,+ kernel_size: int,+ stride: int = 1,+ dilation: int = 1,+ groups: int = 1,+ bias: bool = True,+ padding_mode: str = 'zeros',+ device=None,+ dtype=None+ ) -> None:+ super(CausalConv1dDownSample, self).__init__(in_channels, out_channels,+ kernel_size, stride,+ padding=0, dilation=dilation,+ groups=groups, bias=bias,+ padding_mode=padding_mode,+ device=device, dtype=dtype)+ assert stride != 1 and dilation == 1+ assert kernel_size % stride == 0+ self.causal_padding = stride - 1++ def forward(self, x: torch.Tensor, cache: torch.Tensor = torch.zeros(0, 0, 0)) -> Tuple[torch.Tensor, torch.Tensor]:+ if cache.size(2) == 0:+ x = F.pad(x, (self.causal_padding, 0), value=0.0)+ else:+ assert cache.size(2) == self.causal_padding+ x = torch.concat([cache, x], dim=2)+ x = super(CausalConv1dDownSample, self).forward(x)+ return x+++class CausalConv1dUpsample(torch.nn.Conv1d):+ def __init__(+ self,+ in_channels: int,+ out_channels: int,+ kernel_size: int,+ stride: int = 1,+ dilation: int = 1,+ groups: int = 1,+ bias: bool = True,+ padding_mode: str = 'zeros',+ device=None,+ dtype=None+ ) -> None:+ super(CausalConv1dUpsample, self).__init__(in_channels, out_channels,+ kernel_size, 1,+ padding=0, dilation=dilation,+ groups=groups, bias=bias,+ padding_mode=padding_mode,+ device=device, dtype=dtype)+ assert dilation == 1+ self.causal_padding = kernel_size - 1+ self.upsample = torch.nn.Upsample(scale_factor=stride, mode='nearest')++ def forward(self, x: torch.Tensor, cache: torch.Tensor = torch.zeros(0, 0, 0)) -> Tuple[torch.Tensor, torch.Tensor]:+ x = self.upsample(x)+ input_timestep = x.shape[2]+ if cache.size(2) == 0:+ x = F.pad(x, (self.causal_padding, 0), value=0.0)+ else:+ assert cache.size(2) == self.causal_padding+ x = torch.concat([cache, x], dim=2)+ x = super(CausalConv1dUpsample, self).forward(x)+ assert input_timestep == x.shape[2]+ return xdiff --git a/freevideo_engine/dubbing/cosyvoice3/transformer/upsample_encoder.py b/freevideo_engine/dubbing/cosyvoice3/transformer/upsample_encoder.pynew file mode 100644index 0000000..4067e8e--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/transformer/upsample_encoder.py@@ -0,0 +1,60 @@+# Copyright (c) 2021 Mobvoi Inc (Binbin Zhang, Di Wu)+# 2022 Xingchen Song ([email protected])+# 2024 Alibaba Inc (Xiang Lyu)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# Modified from ESPnet(https://github.com/espnet/espnet)+# Modified by FreeVideo: only PreLookaheadLayer is kept.+import torch+from torch import nn+from torch.nn import functional as F+++class PreLookaheadLayer(nn.Module):+ def __init__(self, in_channels: int, channels: int, pre_lookahead_len: int = 1):+ super().__init__()+ self.in_channels = in_channels+ self.channels = channels+ self.pre_lookahead_len = pre_lookahead_len+ self.conv1 = nn.Conv1d(+ in_channels, channels,+ kernel_size=pre_lookahead_len + 1,+ stride=1, padding=0,+ )+ self.conv2 = nn.Conv1d(+ channels, in_channels,+ kernel_size=3, stride=1, padding=0,+ )++ def forward(self, inputs: torch.Tensor, context: torch.Tensor = torch.zeros(0, 0, 0)) -> torch.Tensor:+ """+ inputs: (batch_size, seq_len, channels)+ """+ outputs = inputs.transpose(1, 2).contiguous()+ context = context.transpose(1, 2).contiguous()+ # look ahead+ if context.size(2) == 0:+ outputs = F.pad(outputs, (0, self.pre_lookahead_len), mode='constant', value=0.0)+ else:+ assert self.training is False, 'you have passed context, make sure that you are running inference mode'+ assert context.size(2) == self.pre_lookahead_len+ outputs = F.pad(torch.concat([outputs, context], dim=2), (0, self.pre_lookahead_len - context.size(2)), mode='constant', value=0.0)+ outputs = F.leaky_relu(self.conv1(outputs))+ # outputs+ outputs = F.pad(outputs, (self.conv2.kernel_size[0] - 1, 0), mode='constant', value=0.0)+ outputs = self.conv2(outputs)+ outputs = outputs.transpose(1, 2).contiguous()++ # residual connection+ outputs = outputs + inputs+ return outputsdiff --git a/freevideo_engine/dubbing/cosyvoice3/utils/__init__.py b/freevideo_engine/dubbing/cosyvoice3/utils/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/freevideo_engine/dubbing/cosyvoice3/utils/common.py b/freevideo_engine/dubbing/cosyvoice3/utils/common.pynew file mode 100644index 0000000..3a8ad7a--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/utils/common.py@@ -0,0 +1,167 @@+# Copyright (c) 2020 Mobvoi Inc (Binbin Zhang)+# 2024 Alibaba Inc (authors: Xiang Lyu)+# 2025 Alibaba Inc (authors: Xiang Lyu, Bofan Zhou)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# Modified from ESPnet(https://github.com/espnet/espnet)+"""Unility functions for Transformer."""++import random+from typing import List++import numpy as np+import torch++IGNORE_ID = -1++def pad_list(xs: List[torch.Tensor], pad_value: int):+ """Perform padding for the list of tensors.++ Args:+ xs (List): List of Tensors [(T_1, `*`), (T_2, `*`), ..., (T_B, `*`)].+ pad_value (float): Value for padding.++ Returns:+ Tensor: Padded tensor (B, Tmax, `*`).++ Examples:+ >>> x = [torch.ones(4), torch.ones(2), torch.ones(1)]+ >>> x+ [tensor([1., 1., 1., 1.]), tensor([1., 1.]), tensor([1.])]+ >>> pad_list(x, 0)+ tensor([[1., 1., 1., 1.],+ [1., 1., 0., 0.],+ [1., 0., 0., 0.]])++ """+ max_len = max([len(item) for item in xs])+ batchs = len(xs)+ ndim = xs[0].ndim+ if ndim == 1:+ pad_res = torch.zeros(batchs,+ max_len,+ dtype=xs[0].dtype,+ device=xs[0].device)+ elif ndim == 2:+ pad_res = torch.zeros(batchs,+ max_len,+ xs[0].shape[1],+ dtype=xs[0].dtype,+ device=xs[0].device)+ elif ndim == 3:+ pad_res = torch.zeros(batchs,+ max_len,+ xs[0].shape[1],+ xs[0].shape[2],+ dtype=xs[0].dtype,+ device=xs[0].device)+ else:+ raise ValueError(f"Unsupported ndim: {ndim}")+ pad_res.fill_(pad_value)+ for i in range(batchs):+ pad_res[i, :len(xs[i])] = xs[i]+ return pad_res+++def th_accuracy(pad_outputs: torch.Tensor, pad_targets: torch.Tensor,+ ignore_label: int) -> torch.Tensor:+ """Calculate accuracy.++ Args:+ pad_outputs (Tensor): Prediction tensors (B * Lmax, D).+ pad_targets (LongTensor): Target label tensors (B, Lmax).+ ignore_label (int): Ignore label id.++ Returns:+ torch.Tensor: Accuracy value (0.0 - 1.0).++ """+ pad_pred = pad_outputs.view(pad_targets.size(0), pad_targets.size(1),+ pad_outputs.size(1)).argmax(2)+ mask = pad_targets != ignore_label+ numerator = torch.sum(+ pad_pred.masked_select(mask) == pad_targets.masked_select(mask))+ denominator = torch.sum(mask)+ return (numerator / denominator).detach()+++def get_padding(kernel_size, dilation=1):+ return int((kernel_size * dilation - dilation) / 2)+++def init_weights(m, mean=0.0, std=0.01):+ classname = m.__class__.__name__+ if classname.find("Conv") != -1:+ m.weight.data.normal_(mean, std)+++# Repetition Aware Sampling in VALL-E 2+def ras_sampling(weighted_scores, decoded_tokens, sampling, top_p=0.8, top_k=25, win_size=10, tau_r=0.1):+ top_ids = nucleus_sampling(weighted_scores, top_p=top_p, top_k=top_k)+ rep_num = (torch.tensor(decoded_tokens[-win_size:]).to(weighted_scores.device) == top_ids).sum().item()+ if rep_num >= win_size * tau_r:+ weighted_scores[top_ids] = -float('inf')+ top_ids = random_sampling(weighted_scores, decoded_tokens, sampling)+ return top_ids+++def nucleus_sampling(weighted_scores, top_p=0.8, top_k=25):+ prob, indices = [], []+ cum_prob = 0.0+ sorted_value, sorted_idx = weighted_scores.softmax(dim=0).sort(descending=True, stable=True)+ for i in range(len(sorted_idx)):+ # sampling both top-p and numbers.+ if cum_prob < top_p and len(prob) < top_k:+ cum_prob += sorted_value[i]+ prob.append(sorted_value[i])+ indices.append(sorted_idx[i])+ else:+ break+ prob = torch.tensor(prob).to(weighted_scores)+ indices = torch.tensor(indices, dtype=torch.long).to(weighted_scores.device)+ top_ids = indices[prob.multinomial(1, replacement=True)].item()+ return top_ids+++def random_sampling(weighted_scores, decoded_tokens, sampling):+ top_ids = weighted_scores.softmax(dim=0).multinomial(1, replacement=True).item()+ return top_ids+++def fade_in_out(fade_in_mel, fade_out_mel, window):+ device = fade_in_mel.device+ fade_in_mel, fade_out_mel = fade_in_mel.cpu(), fade_out_mel.cpu()+ mel_overlap_len = int(window.shape[0] / 2)+ if fade_in_mel.device == torch.device('cpu'):+ fade_in_mel = fade_in_mel.clone()+ fade_in_mel[..., :mel_overlap_len] = fade_in_mel[..., :mel_overlap_len] * window[:mel_overlap_len] + \+ fade_out_mel[..., -mel_overlap_len:] * window[mel_overlap_len:]+ return fade_in_mel.to(device)+++def set_all_random_seed(seed):+ random.seed(seed)+ np.random.seed(seed)+ torch.manual_seed(seed)+ torch.cuda.manual_seed_all(seed)+++def mask_to_bias(mask: torch.Tensor, dtype: torch.dtype) -> torch.Tensor:+ assert mask.dtype == torch.bool+ assert dtype in [torch.float32, torch.bfloat16, torch.float16]+ mask = mask.to(dtype)+ # attention mask bias+ # NOTE(Mddct): torch.finfo jit issues+ # chunk_masks = (1.0 - chunk_masks) * torch.finfo(dtype).min+ mask = (1.0 - mask) * -1.0e+10+ return maskdiff --git a/freevideo_engine/dubbing/cosyvoice3/utils/file_utils.py b/freevideo_engine/dubbing/cosyvoice3/utils/file_utils.pynew file mode 100644index 0000000..e9a6e36--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/utils/file_utils.py@@ -0,0 +1,32 @@+# Copyright (c) 2021 Mobvoi Inc. (authors: Binbin Zhang)+# 2024 Alibaba Inc (authors: Xiang Lyu, Zetao Hu)+# 2025 Alibaba Inc (authors: Xiang Lyu, Yabin Li)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# Modified by FreeVideo: only load_wav is kept.++import logging++import torch+import torchaudio+++def load_wav(wav, target_sr, min_sr=16000):+ from freevideo_engine.dubbing import wavio+ data, sample_rate = wavio.read(wav)+ speech = torch.from_numpy(data.T.copy())+ speech = speech.mean(dim=0, keepdim=True)+ if sample_rate != target_sr:+ assert sample_rate >= min_sr, 'wav sample rate {} must be greater than {}'.format(sample_rate, target_sr)+ speech = torchaudio.transforms.Resample(orig_freq=sample_rate, new_freq=target_sr)(speech)+ return speechdiff --git a/freevideo_engine/dubbing/cosyvoice3/utils/frontend_utils.py b/freevideo_engine/dubbing/cosyvoice3/utils/frontend_utils.pynew file mode 100644index 0000000..ea1c9fc--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/utils/frontend_utils.py@@ -0,0 +1,136 @@+# Copyright (c) 2024 Alibaba Inc (authors: Xiang Lyu, Zhihao Du)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++import re+import regex+chinese_char_pattern = re.compile(r'[\u4e00-\u9fff]+')+++# whether contain chinese character+def contains_chinese(text):+ return bool(chinese_char_pattern.search(text))+++# replace special symbol+def replace_corner_mark(text):+ text = text.replace('²', '平方')+ text = text.replace('³', '立方')+ return text+++# remove meaningless symbol+def remove_bracket(text):+ text = text.replace('(', '').replace(')', '')+ text = text.replace('【', '').replace('】', '')+ text = text.replace('`', '').replace('`', '')+ text = text.replace("——", " ")+ return text+++# spell Arabic numerals+def spell_out_number(text: str, inflect_parser):+ new_text = []+ st = None+ for i, c in enumerate(text):+ if not c.isdigit():+ if st is not None:+ num_str = inflect_parser.number_to_words(text[st: i])+ new_text.append(num_str)+ st = None+ new_text.append(c)+ else:+ if st is None:+ st = i+ if st is not None and st < len(text):+ num_str = inflect_parser.number_to_words(text[st:])+ new_text.append(num_str)+ return ''.join(new_text)+++# split paragrah logic:+# 1. per sentence max len token_max_n, min len token_min_n, merge if last sentence len less than merge_len+# 2. cal sentence len according to lang+# 3. split sentence according to puncatation+def split_paragraph(text: str, tokenize, lang="zh", token_max_n=80, token_min_n=60, merge_len=20, comma_split=False):+ def calc_utt_length(_text: str):+ if lang == "zh":+ return len(_text)+ else:+ return len(tokenize(_text))++ def should_merge(_text: str):+ if lang == "zh":+ return len(_text) < merge_len+ else:+ return len(tokenize(_text)) < merge_len++ if lang == "zh":+ pounc = ['。', '?', '!', ';', ':', '、', '.', '?', '!', ';']+ else:+ pounc = ['.', '?', '!', ';', ':']+ if comma_split:+ pounc.extend([',', ','])++ if text[-1] not in pounc:+ if lang == "zh":+ text += "。"+ else:+ text += "."++ st = 0+ utts = []+ for i, c in enumerate(text):+ if c in pounc:+ if len(text[st: i]) > 0:+ utts.append(text[st: i] + c)+ if i + 1 < len(text) and text[i + 1] in ['"', '”']:+ tmp = utts.pop(-1)+ utts.append(tmp + text[i + 1])+ st = i + 2+ else:+ st = i + 1++ final_utts = []+ cur_utt = ""+ for utt in utts:+ if calc_utt_length(cur_utt + utt) > token_max_n and calc_utt_length(cur_utt) > token_min_n:+ final_utts.append(cur_utt)+ cur_utt = ""+ cur_utt = cur_utt + utt+ if len(cur_utt) > 0:+ if should_merge(cur_utt) and len(final_utts) != 0:+ final_utts[-1] = final_utts[-1] + cur_utt+ else:+ final_utts.append(cur_utt)++ return final_utts+++# remove blank between chinese character+def replace_blank(text: str):+ out_str = []+ for i, c in enumerate(text):+ if c == " ":+ if ((text[i + 1].isascii() and text[i + 1] != " ") and+ (text[i - 1].isascii() and text[i - 1] != " ")):+ out_str.append(c)+ else:+ out_str.append(c)+ return "".join(out_str)+++def is_only_punctuation(text):+ # Regular expression: Match strings that consist only of punctuation marks or are empty.+ punctuation_pattern = r'^[\p{P}\p{S}]*$'+ return bool(regex.fullmatch(punctuation_pattern, text))diff --git a/freevideo_engine/dubbing/cosyvoice3/utils/mask.py b/freevideo_engine/dubbing/cosyvoice3/utils/mask.pynew file mode 100644index 0000000..5d3dfd6--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/utils/mask.py@@ -0,0 +1,265 @@+# Copyright (c) 2019 Shigeki Karita+# 2020 Mobvoi Inc (Binbin Zhang)+# 2024 Alibaba Inc (authors: Xiang Lyu)+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++import torch+'''+def subsequent_mask(+ size: int,+ device: torch.device = torch.device("cpu"),+) -> torch.Tensor:+ """Create mask for subsequent steps (size, size).++ This mask is used only in decoder which works in an auto-regressive mode.+ This means the current step could only do attention with its left steps.++ In encoder, fully attention is used when streaming is not necessary and+ the sequence is not long. In this case, no attention mask is needed.++ When streaming is need, chunk-based attention is used in encoder. See+ subsequent_chunk_mask for the chunk-based attention mask.++ Args:+ size (int): size of mask+ str device (str): "cpu" or "cuda" or torch.Tensor.device+ dtype (torch.device): result dtype++ Returns:+ torch.Tensor: mask++ Examples:+ >>> subsequent_mask(3)+ [[1, 0, 0],+ [1, 1, 0],+ [1, 1, 1]]+ """+ ret = torch.ones(size, size, device=device, dtype=torch.bool)+ return torch.tril(ret)+'''+++def subsequent_mask(+ size: int,+ device: torch.device = torch.device("cpu"),+) -> torch.Tensor:+ """Create mask for subsequent steps (size, size).++ This mask is used only in decoder which works in an auto-regressive mode.+ This means the current step could only do attention with its left steps.++ In encoder, fully attention is used when streaming is not necessary and+ the sequence is not long. In this case, no attention mask is needed.++ When streaming is need, chunk-based attention is used in encoder. See+ subsequent_chunk_mask for the chunk-based attention mask.++ Args:+ size (int): size of mask+ str device (str): "cpu" or "cuda" or torch.Tensor.device+ dtype (torch.device): result dtype++ Returns:+ torch.Tensor: mask++ Examples:+ >>> subsequent_mask(3)+ [[1, 0, 0],+ [1, 1, 0],+ [1, 1, 1]]+ """+ arange = torch.arange(size, device=device)+ mask = arange.expand(size, size)+ arange = arange.unsqueeze(-1)+ mask = mask <= arange+ return mask+++def subsequent_chunk_mask_deprecated(+ size: int,+ chunk_size: int,+ num_left_chunks: int = -1,+ device: torch.device = torch.device("cpu"),+) -> torch.Tensor:+ """Create mask for subsequent steps (size, size) with chunk size,+ this is for streaming encoder++ Args:+ size (int): size of mask+ chunk_size (int): size of chunk+ num_left_chunks (int): number of left chunks+ <0: use full chunk+ >=0: use num_left_chunks+ device (torch.device): "cpu" or "cuda" or torch.Tensor.device++ Returns:+ torch.Tensor: mask++ Examples:+ >>> subsequent_chunk_mask(4, 2)+ [[1, 1, 0, 0],+ [1, 1, 0, 0],+ [1, 1, 1, 1],+ [1, 1, 1, 1]]+ """+ ret = torch.zeros(size, size, device=device, dtype=torch.bool)+ for i in range(size):+ if num_left_chunks < 0:+ start = 0+ else:+ start = max((i // chunk_size - num_left_chunks) * chunk_size, 0)+ ending = min((i // chunk_size + 1) * chunk_size, size)+ ret[i, start:ending] = True+ return ret+++def subsequent_chunk_mask(+ size: int,+ chunk_size: int,+ num_left_chunks: int = -1,+ device: torch.device = torch.device("cpu"),+) -> torch.Tensor:+ """Create mask for subsequent steps (size, size) with chunk size,+ this is for streaming encoder++ Args:+ size (int): size of mask+ chunk_size (int): size of chunk+ num_left_chunks (int): number of left chunks+ <0: use full chunk+ >=0: use num_left_chunks+ device (torch.device): "cpu" or "cuda" or torch.Tensor.device++ Returns:+ torch.Tensor: mask++ Examples:+ >>> subsequent_chunk_mask(4, 2)+ [[1, 1, 0, 0],+ [1, 1, 0, 0],+ [1, 1, 1, 1],+ [1, 1, 1, 1]]+ """+ # NOTE this modified implementation meets onnx export requirements, but it doesn't support num_left_chunks+ pos_idx = torch.arange(size, device=device)+ block_value = (torch.div(pos_idx, chunk_size, rounding_mode='trunc') + 1) * chunk_size+ ret = pos_idx.unsqueeze(0) < block_value.unsqueeze(1)+ return ret+++def add_optional_chunk_mask(xs: torch.Tensor,+ masks: torch.Tensor,+ use_dynamic_chunk: bool,+ use_dynamic_left_chunk: bool,+ decoding_chunk_size: int,+ static_chunk_size: int,+ num_decoding_left_chunks: int,+ enable_full_context: bool = True):+ """ Apply optional mask for encoder.++ Args:+ xs (torch.Tensor): padded input, (B, L, D), L for max length+ mask (torch.Tensor): mask for xs, (B, 1, L)+ use_dynamic_chunk (bool): whether to use dynamic chunk or not+ use_dynamic_left_chunk (bool): whether to use dynamic left chunk for+ training.+ decoding_chunk_size (int): decoding chunk size for dynamic chunk, it's+ 0: default for training, use random dynamic chunk.+ <0: for decoding, use full chunk.+ >0: for decoding, use fixed chunk size as set.+ static_chunk_size (int): chunk size for static chunk training/decoding+ if it's greater than 0, if use_dynamic_chunk is true,+ this parameter will be ignored+ num_decoding_left_chunks: number of left chunks, this is for decoding,+ the chunk size is decoding_chunk_size.+ >=0: use num_decoding_left_chunks+ <0: use all left chunks+ enable_full_context (bool):+ True: chunk size is either [1, 25] or full context(max_len)+ False: chunk size ~ U[1, 25]++ Returns:+ torch.Tensor: chunk mask of the input xs.+ """+ # Whether to use chunk mask or not+ if use_dynamic_chunk:+ max_len = xs.size(1)+ if decoding_chunk_size < 0:+ chunk_size = max_len+ num_left_chunks = -1+ elif decoding_chunk_size > 0:+ chunk_size = decoding_chunk_size+ num_left_chunks = num_decoding_left_chunks+ else:+ # chunk size is either [1, 25] or full context(max_len).+ # Since we use 4 times subsampling and allow up to 1s(100 frames)+ # delay, the maximum frame is 100 / 4 = 25.+ chunk_size = torch.randint(1, max_len, (1, )).item()+ num_left_chunks = -1+ if chunk_size > max_len // 2 and enable_full_context:+ chunk_size = max_len+ else:+ chunk_size = chunk_size % 25 + 1+ if use_dynamic_left_chunk:+ max_left_chunks = (max_len - 1) // chunk_size+ num_left_chunks = torch.randint(0, max_left_chunks,+ (1, )).item()+ chunk_masks = subsequent_chunk_mask(xs.size(1), chunk_size,+ num_left_chunks,+ xs.device) # (L, L)+ chunk_masks = chunk_masks.unsqueeze(0) # (1, L, L)+ chunk_masks = masks & chunk_masks # (B, L, L)+ elif static_chunk_size > 0:+ num_left_chunks = num_decoding_left_chunks+ chunk_masks = subsequent_chunk_mask(xs.size(1), static_chunk_size,+ num_left_chunks,+ xs.device) # (L, L)+ chunk_masks = chunk_masks.unsqueeze(0) # (1, L, L)+ chunk_masks = masks & chunk_masks # (B, L, L)+ else:+ chunk_masks = masks+ assert chunk_masks.dtype == torch.bool+ if (chunk_masks.sum(dim=-1) == 0).sum().item() != 0:+ print('get chunk_masks all false at some timestep, force set to true, make sure they are masked in futuer computation!')+ chunk_masks[chunk_masks.sum(dim=-1) == 0] = True+ return chunk_masks+++def make_pad_mask(lengths: torch.Tensor, max_len: int = 0) -> torch.Tensor:+ """Make mask tensor containing indices of padded part.++ See description of make_non_pad_mask.++ Args:+ lengths (torch.Tensor): Batch of lengths (B,).+ Returns:+ torch.Tensor: Mask tensor containing indices of padded part.++ Examples:+ >>> lengths = [5, 3, 2]+ >>> make_pad_mask(lengths)+ masks = [[0, 0, 0, 0 ,0],+ [0, 0, 0, 1, 1],+ [0, 0, 1, 1, 1]]+ """+ batch_size = lengths.size(0)+ max_len = max_len if max_len > 0 else lengths.max().item()+ seq_range = torch.arange(0,+ max_len,+ dtype=torch.int64,+ device=lengths.device)+ seq_range_expand = seq_range.unsqueeze(0).expand(batch_size, max_len)+ seq_length_expand = lengths.unsqueeze(-1)+ mask = seq_range_expand >= seq_length_expand+ return maskdiff --git a/freevideo_engine/dubbing/cosyvoice3/utils/mel.py b/freevideo_engine/dubbing/cosyvoice3/utils/mel.pynew file mode 100644index 0000000..bca293d--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3/utils/mel.py@@ -0,0 +1,98 @@+import numpy as np+import torch+++def _hz_to_mel(f):+ f = np.asarray(f, dtype=np.float64)+ lin = f / (200.0 / 3)+ log = 15.0 + np.log(np.maximum(f, 1e-10) / 1000.0) / (np.log(6.4) / 27.0)+ return np.where(f >= 1000.0, log, lin)+++def _mel_to_hz(m):+ m = np.asarray(m, dtype=np.float64)+ lin = (200.0 / 3) * m+ log = 1000.0 * np.exp((np.log(6.4) / 27.0) * (m - 15.0))+ return np.where(m >= 15.0, log, lin)+++def librosa_mel_fn(sr, n_fft, n_mels, fmin=0.0, fmax=None):+ """Slaney-scale, Slaney-normalised mel filterbank (same result as librosa.filters.mel, float32)."""+ fmax = sr / 2.0 if fmax is None else fmax+ freqs = np.linspace(0.0, sr / 2.0, n_fft // 2 + 1)+ pts = _mel_to_hz(np.linspace(_hz_to_mel(fmin), _hz_to_mel(fmax), n_mels + 2))+ fdiff = np.diff(pts)+ ramps = pts[:, None] - freqs[None, :]+ lower = -ramps[:-2] / fdiff[:-1, None]+ upper = ramps[2:] / fdiff[1:, None]+ w = np.maximum(0.0, np.minimum(lower, upper))+ w *= (2.0 / (pts[2:n_mels + 2] - pts[:n_mels]))[:, None]+ return w.astype(np.float32)++_FILTERS = {}+++def log_mel_spectrogram(audio, n_mels=128, n_fft=400, hop=160, sr=16000):+ """Whisper-compatible log-mel spectrogram (Slaney filters, log10, floor at max - 8, scaled to ~[-1, 1])."""+ key = (n_mels, n_fft, sr, audio.device)+ if key not in _FILTERS:+ _FILTERS[key] = torch.from_numpy(librosa_mel_fn(sr, n_fft, n_mels)).to(audio.device)+ window = torch.hann_window(n_fft).to(audio.device)+ stft = torch.stft(audio, n_fft, hop, window=window, return_complex=True)+ magnitudes = stft[..., :-1].abs() ** 2+ log_spec = torch.clamp(_FILTERS[key] @ magnitudes, min=1e-10).log10()+ log_spec = torch.maximum(log_spec, log_spec.max() - 8.0)+ return (log_spec + 4.0) / 4.0+++# mel_spectrogram below is copied from Matcha-TTS matcha/utils/audio.py (MIT; derived from HiFi-GAN, MIT)+def dynamic_range_compression_torch(x, C=1, clip_val=1e-5):+ return torch.log(torch.clamp(x, min=clip_val) * C)+++def spectral_normalize_torch(magnitudes):+ return dynamic_range_compression_torch(magnitudes)+++mel_basis = {}+hann_window = {}+++def mel_spectrogram(y, n_fft, num_mels, sampling_rate, hop_size, win_size, fmin, fmax, center=False):+ if torch.min(y) < -1.0:+ print("min value is ", torch.min(y))+ if torch.max(y) > 1.0:+ print("max value is ", torch.max(y))++ global mel_basis, hann_window # pylint: disable=global-statement,global-variable-not-assigned+ if f"{str(fmax)}_{str(y.device)}" not in mel_basis:+ mel = librosa_mel_fn(sr=sampling_rate, n_fft=n_fft, n_mels=num_mels, fmin=fmin, fmax=fmax)+ mel_basis[str(fmax) + "_" + str(y.device)] = torch.from_numpy(mel).float().to(y.device)+ hann_window[str(y.device)] = torch.hann_window(win_size).to(y.device)++ y = torch.nn.functional.pad(+ y.unsqueeze(1), (int((n_fft - hop_size) / 2), int((n_fft - hop_size) / 2)), mode="reflect"+ )+ y = y.squeeze(1)++ spec = torch.view_as_real(+ torch.stft(+ y,+ n_fft,+ hop_length=hop_size,+ win_length=win_size,+ window=hann_window[str(y.device)],+ center=center,+ pad_mode="reflect",+ normalized=False,+ onesided=True,+ return_complex=True,+ )+ )++ spec = torch.sqrt(spec.pow(2).sum(-1) + (1e-9))++ spec = torch.matmul(mel_basis[str(fmax) + "_" + str(y.device)], spec)+ spec = spectral_normalize_torch(spec)++ return specdiff --git a/freevideo_engine/dubbing/cosyvoice3_pack.json b/freevideo_engine/dubbing/cosyvoice3_pack.jsonnew file mode 100644index 0000000..9becf95--- /dev/null+++ b/freevideo_engine/dubbing/cosyvoice3_pack.json@@ -0,0 +1,159 @@+{+ "format": 1,+ "id": "cosyvoice3",+ "name": "Fun-CosyVoice3-0.5B-2512",+ "license": "Apache-2.0",+ "source": "https://github.com/FunAudioLLM/CosyVoice",+ "modelscope": {+ "repo": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",+ "revision": "master"+ },+ "huggingface": {+ "repo": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512"+ },+ "files": [+ {+ "file": "llm.pt",+ "bytes": 2024669519,+ "sha256": "69f43bd545131c30e98947fb360ea8b4dc9916d8e83dded7757c7ea4f5a24970"+ },+ {+ "file": "flow.pt",+ "bytes": 1329116148,+ "sha256": "a6fab32a7825e5b0bc855ddd948f8db9370b0a786fbc249caa4595e95b608e4b"+ },+ {+ "file": "hift.pt",+ "bytes": 83202622,+ "sha256": "b279d7641eb97ae55b3b540cfba4f953c26492a2df758328a89a4d007ab87a65"+ },+ {+ "file": "campplus.onnx",+ "bytes": 28303423,+ "sha256": "a6ac6a63997761ae2997373e2ee1c47040854b4b759ea41ec48e4e42df0f4d73"+ },+ {+ "file": "speech_tokenizer_v3.onnx",+ "bytes": 969451503,+ "sha256": "23236a74175dbdda47afc66dbadd5bcb41303c467a57c261cb8539ad9db9208d"+ },+ {+ "file": "CosyVoice-BlankEN/config.json",+ "bytes": 659,+ "sha256": "168aa1bd401abc3bc262ba15ba4e499627a8b4e006e9d050b47c22de20660185"+ },+ {+ "file": "CosyVoice-BlankEN/generation_config.json",+ "bytes": 242,+ "sha256": "e558847a8b4402616f1273797b015104dc266fe4b520056fca88823ba8f8ebe6"+ },+ {+ "file": "CosyVoice-BlankEN/merges.txt",+ "bytes": 1402109,+ "sha256": "ac8ff86a72bee70828fbc1119bc4398c6f3a9a6e490d7b0dbe917be025478bd0"+ },+ {+ "file": "CosyVoice-BlankEN/tokenizer_config.json",+ "bytes": 1287,+ "sha256": "482bd979881423375ca5414e4e0d94cd7c5349dbb17fffd46b4d36d71e62a1bc"+ },+ {+ "file": "CosyVoice-BlankEN/vocab.json",+ "bytes": 2776833,+ "sha256": "ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910"+ }+ ],+ "runtime": {+ "python": "cp312",+ "note": "Hash-locked PyPI wheels, downloaded when the voice pack is installed; not part of the installer.",+ "wheels": {+ "linux-x86_64": [+ {+ "name": "onnxruntime",+ "version": "1.31.1",+ "license": "MIT",+ "file": "onnxruntime-1.31.1-cp312-cp312-manylinux_2_28_x86_64.whl",+ "bytes": 23760988,+ "sha256": "1f3dea47f2f0e5117c2404cea7d21fd253ae7836a3bf167a501cdfbb2600d536",+ "path": "eb/27/c6a823bac55b2977bea2027c3ec9393a2c1002cb9a09eb40354563e02730/onnxruntime-1.31.1-cp312-cp312-manylinux_2_28_x86_64.whl"+ },+ {+ "name": "flatbuffers",+ "version": "25.12.19",+ "license": "Apache-2.0",+ "file": "flatbuffers-25.12.19-py2.py3-none-any.whl",+ "bytes": 26661,+ "sha256": "7634f50c427838bb021c2d66a3d1168e9d199b0607e6329399f04846d42e20b4",+ "path": "e8/2d/d2a548598be01649e2d46231d151a6c56d10b964d94043a335ae56ea2d92/flatbuffers-25.12.19-py2.py3-none-any.whl"+ },+ {+ "name": "protobuf",+ "version": "7.36.2",+ "license": "BSD-3-Clause",+ "file": "protobuf-7.36.2-cp310-abi3-manylinux2014_x86_64.whl",+ "bytes": 343223,+ "sha256": "89f23aa53c24553a2416fd4fd1ec06f74fa42b14b546d8883128813f775bbfd2",+ "path": "db/f3/3996583dd2906297a637af12114deddf7658af6e683fedb83be061983fb5/protobuf-7.36.2-cp310-abi3-manylinux2014_x86_64.whl"+ }+ ],+ "windows-x64": [+ {+ "name": "onnxruntime",+ "version": "1.31.1",+ "license": "MIT",+ "file": "onnxruntime-1.31.1-cp312-cp312-win_amd64.whl",+ "bytes": 14888466,+ "sha256": "ae8a9a351efda685cc59a66226cdefaa37a126c1d66872ca23a2de9425326130",+ "path": "45/b2/793b4051d8dc6acb379ead9b72050f0ffa283ca49a6904add2e218b83e2e/onnxruntime-1.31.1-cp312-cp312-win_amd64.whl"+ },+ {+ "name": "flatbuffers",+ "version": "25.12.19",+ "license": "Apache-2.0",+ "file": "flatbuffers-25.12.19-py2.py3-none-any.whl",+ "bytes": 26661,+ "sha256": "7634f50c427838bb021c2d66a3d1168e9d199b0607e6329399f04846d42e20b4",+ "path": "e8/2d/d2a548598be01649e2d46231d151a6c56d10b964d94043a335ae56ea2d92/flatbuffers-25.12.19-py2.py3-none-any.whl"+ },+ {+ "name": "protobuf",+ "version": "7.36.2",+ "license": "BSD-3-Clause",+ "file": "protobuf-7.36.2-cp310-abi3-win_amd64.whl",+ "bytes": 456514,+ "sha256": "a300819d441e078a5608c0d3c709796bb548136058fda017ae51d425b44fd353",+ "path": "8a/55/b77bda4e5e5f5971fb51b07663694690e9afdb9402136c16a522bd621cad/protobuf-7.36.2-cp310-abi3-win_amd64.whl"+ }+ ],+ "macos-arm64": [+ {+ "name": "onnxruntime",+ "version": "1.31.1",+ "license": "MIT",+ "file": "onnxruntime-1.31.1-cp312-cp312-macosx_14_0_arm64.whl",+ "bytes": 20882125,+ "sha256": "96dd2564b10ea7153ad9c207e7cd8e5917cb3ba521e0e0a62ac03486fdc8d56c",+ "path": "0e/f7/cecc0efeeea83bff2f7789ae1b80f27fa0f0210d6d9d66d1f0031a6b5e47/onnxruntime-1.31.1-cp312-cp312-macosx_14_0_arm64.whl"+ },+ {+ "name": "flatbuffers",+ "version": "25.12.19",+ "license": "Apache-2.0",+ "file": "flatbuffers-25.12.19-py2.py3-none-any.whl",+ "bytes": 26661,+ "sha256": "7634f50c427838bb021c2d66a3d1168e9d199b0607e6329399f04846d42e20b4",+ "path": "e8/2d/d2a548598be01649e2d46231d151a6c56d10b964d94043a335ae56ea2d92/flatbuffers-25.12.19-py2.py3-none-any.whl"+ },+ {+ "name": "protobuf",+ "version": "7.36.2",+ "license": "BSD-3-Clause",+ "file": "protobuf-7.36.2-cp310-abi3-macosx_10_9_universal2.whl",+ "bytes": 456039,+ "sha256": "cbc70b17ee27e28894c7fee8bb04be1abead49e936bc70eb60052531eee2079e",+ "path": "32/72/98342feb672507c8f3a69e34b4fa8961f608edba5c1a48a6f47156d92cb5/protobuf-7.36.2-cp310-abi3-macosx_10_9_universal2.whl"+ }+ ]+ }+ }+}diff --git a/freevideo_engine/dubbing/pack.py b/freevideo_engine/dubbing/pack.pynew file mode 100644index 0000000..5154869--- /dev/null+++ b/freevideo_engine/dubbing/pack.py@@ -0,0 +1,365 @@+"""On-demand download of the dubbing model (Fun-CosyVoice3-0.5B-2512) and its inference runtime.++Nothing here ships in the installer. The catalog (`cosyvoice3_pack.json`) lists every file with its+size and sha256; a file is used only after it matched. Layout under `<model_root>/dubbing/`:+`cosyvoice3/` (the weights), `wheels/` (the runtime wheels as downloaded), `runtime/<id>/` (the wheels+unpacked; put on `sys.path` before `import onnxruntime`) and `licenses/` (the runtime licence texts).+ModelScope has no commit that matches a Hugging Face one, so every source is accepted only when the+size and SHA-256 in the catalog match, and the file names are the same on all of them. The runtime+wheels are the PyPI files, fetched from PyPI or its domestic mirror by the same rule. onnxruntime also+requires packaging and sympy; both come with torch in the unified environment, so they are not listed.+The wheels are chosen for the Python of the service process, which is the pinned 3.12 of the product's+own environment. A user's own ComfyUI Python of another version would fail the import in the worker,+which reports it as a general failure.+"""+import hashlib+import json+import os+from pathlib import Path+import shutil+import sys+import zipfile++from .. import disk_space++FAMILY = 'image-models'+RESERVE = 256 * 2**20+LEDGER = 'download-verified.json'+NAME = 'cosyvoice3'+COMPLETE = 'complete.json'+MAX_UNPACKED = 256 * 2**20+LICENSES = 'RUNTIME-LICENSES.txt'+# The two PyPI hosts of the `pypi` download family; a wheel's `path` is the same on both.+PYPI_FILES = {'official': 'https://files.pythonhosted.org/packages/', 'tuna': 'https://pypi.tuna.tsinghua.edu.cn/packages/'}+MODULES = (('onnxruntime', '__init__.py'), ('flatbuffers', '__init__.py'), ('google', 'protobuf', '__init__.py'))+++class PackError(Exception):+ def __init__(self, reason):+ super().__init__(reason)+ self.reason = reason+++def catalog():+ """The file list shipped beside this file; a frozen launcher's bundled copy answers when it is gone."""+ name = 'cosyvoice3_pack.json'+ paths = [Path(__file__).with_name(name)]+ bundle = getattr(sys, '_MEIPASS', None)+ if bundle:+ paths.append(Path(bundle) / 'engine-source' / 'freevideo_engine' / 'dubbing' / name)+ for path in paths:+ try:+ return json.loads(path.read_text(encoding='utf-8'))+ except (OSError, ValueError) as error:+ missing = error+ raise missing+++def rows(spec=None):+ return list((spec or catalog())['files'])+++def base(root=None):+ if root is None:+ from ..paths import model_root+ root = model_root()+ return Path(root) / 'dubbing'+++def directory(root=None):+ return base(root) / NAME+++def _ledger(folder):+ try:+ value = json.loads((folder / LEDGER).read_text(encoding='utf-8'))+ return value if isinstance(value, dict) else {}+ except (OSError, ValueError):+ return {}+++def _intact(folder, row, ledger):+ """Size and the stamp recorded after the last hash match; no hashing here."""+ from .. import provision+ path = folder / row['file']+ try:+ if not path.is_file() or path.stat().st_size != row['bytes']:+ return False+ prior = ledger.get(str(path))+ return isinstance(prior, dict) and all(+ prior.get(key) == value for key, value in provision.file_identity(path, row).items())+ except OSError:+ return False+++def wheel_rows(spec=None):+ """This machine's runtime wheels, or None (other platform or Python, or none listed)."""+ from ..subtitle_pack import platform_key+ runtime = (spec or catalog()).get('runtime') or {}+ if runtime.get('python') != 'cp%d%d' % sys.version_info[:2]:+ return None+ return list((runtime.get('wheels') or {}).get(platform_key()) or []) or None+++def runtime_id(wheels):+ return hashlib.sha256('\n'.join(row['sha256'] for row in wheels).encode('ascii')).hexdigest()[:12]+++def runtime_folder(root=None, spec=None):+ wheels = wheel_rows(spec)+ return base(root) / 'runtime' / runtime_id(wheels) if wheels else None+++def _unpacked(root=None, spec=None):+ folder = runtime_folder(root, spec)+ if folder is None:+ return False+ try:+ receipt = json.loads((folder / COMPLETE).read_text(encoding='utf-8'))+ return receipt.get('id') == folder.name and all(folder.joinpath(*parts).is_file() for parts in MODULES)+ except (OSError, ValueError, KeyError, TypeError):+ return False+++def runtime_path(root=None):+ """The folder to put on sys.path for onnxruntime, or None while it is not unpacked."""+ try:+ return str(runtime_folder(root)) if _unpacked(root) else None+ except (OSError, ValueError, KeyError, TypeError):+ return None+++def activate(path):+ """Put an unpacked runtime on sys.path (once); Windows also needs its DLL folders."""+ if not path:+ return False+ folder = Path(path)+ if str(folder) not in sys.path:+ sys.path.insert(0, str(folder))+ if sys.platform == 'win32' and hasattr(os, 'add_dll_directory'):+ for item in {folder} | {found.parent for found in folder.rglob('*.dll')}:+ os.add_dll_directory(str(item))+ return True+++def ready(root=None, spec=None):+ """Cheap and safe on a UI thread: sizes and the identity stamps of the last content check."""+ try:+ spec = spec or catalog()+ folder = directory(root)+ ledger = _ledger(folder)+ return _unpacked(root, spec) and all(_intact(folder, row, ledger) for row in rows(spec))+ except (OSError, ValueError, KeyError, TypeError):+ return False+++def settle(root=None, spec=None):+ """Same answer, but a file that moved without a stamp (a copied folder) is hashed here once.+ For request handlers, never for a UI thread or under a lock."""+ if ready(root, spec):+ return True+ try:+ from .. import provision+ from ..monitoring import save+ spec = spec or catalog()+ folder = directory(root)+ if not _unpacked(root, spec) or not folder.is_dir():+ return False+ ledger = _ledger(folder)+ found = all(provision.verified(folder / row['file'], row, ledger) for row in rows(spec))+ if found:+ save(folder / LEDGER, ledger)+ return found+ except (OSError, ValueError, KeyError, TypeError):+ return False+++def missing_bytes(root=None, spec=None):+ spec = spec or catalog()+ folder = directory(root)+ ledger = _ledger(folder)+ weights = sum(row['bytes'] for row in rows(spec) if not _intact(folder, row, ledger))+ return weights + (0 if _unpacked(root, spec) else sum(row['bytes'] for row in wheel_rows(spec) or ()))+++def status(root=None):+ """What a page may show: the size of the download and whether the model is in place."""+ spec = catalog()+ wheels = wheel_rows(spec)+ total = sum(row['bytes'] for row in rows(spec)) + sum(row['bytes'] for row in wheels or ())+ return dict(name=NAME, license=spec.get('license'), bytes=missing_bytes(root, spec), total=total,+ ready=ready(root, spec), supported=wheels is not None,+ published=bool(spec.get('modelscope') or spec.get('huggingface')))+++def remote(spec, row):+ """One catalog file with every source FreeVideo may fetch it from (ModelScope, Hugging Face, mirror)."""+ from ..network import source_url+ sources = []+ mirror = spec.get('modelscope')+ if mirror:+ sources.append(dict(id='modelscope', url=source_url(FAMILY, 'modelscope') + '/models/%s/resolve/%s/%s' % (+ mirror['repo'], mirror.get('revision', 'master'), row['file'])))+ hub = spec.get('huggingface')+ if hub:+ path = '/%s/resolve/%s/%s' % (hub['repo'], hub.get('revision', 'main'), row['file'])+ sources.append(dict(id='official', url=source_url('models', 'official') + path))+ sources.append(dict(id='hf-mirror', url=source_url('models', 'hf-mirror') + path))+ return dict(repo=(mirror or hub)['repo'], revision=(mirror or hub).get('revision', 'main'), file=row['file'],+ family='image', sources=sources)+++def runtime_plan(plan):+ """The plan for the runtime wheels: the `image-models` list holds the PyPI hosts in the order the+ plan prefers them (PyPI, or its domestic mirror)."""+ from ..network import ordered+ names = [name for name in ordered(plan, 'pypi') if name in PYPI_FILES] or ['official']+ return dict(plan, sources=dict(plan.get('sources', {}), **{FAMILY: [{'id': name} for name in names]}))+++def runtime_remote(row):+ return dict(repo='pypi/' + row['name'], revision=row['version'], file=row['file'], family='image',+ sources=[dict(id=name, url=PYPI_FILES[name] + row['path']) for name in PYPI_FILES])+++def _unpack(root, wheels, check):+ """Extract the verified wheels into runtime/<id>/ and drop older runtimes. A file that two wheels+ both carry, an unsafe name, or too much data fails the whole unpack."""+ folder = runtime_folder(root)+ parent = folder.parent+ parent.mkdir(parents=True, exist_ok=True)+ work = parent / ('.unpack-' + folder.name)+ shutil.rmtree(work, ignore_errors=True)+ total = 0+ try:+ for row in wheels:+ with zipfile.ZipFile(wheels_folder(root) / row['file']) as archive:+ for item in archive.infolist():+ check()+ name = item.filename+ parts = Path(name).parts+ if (name.startswith('/') or '\\' in name or ':' in name or not parts or '..' in parts+ or (item.external_attr >> 16) & 0o170000 == 0o120000):+ raise PackError('failed')+ total += item.file_size+ if total > MAX_UNPACKED:+ raise PackError('failed')+ target = work.joinpath(*parts)+ if item.is_dir():+ target.mkdir(parents=True, exist_ok=True)+ continue+ if target.exists():+ raise PackError('failed')+ target.parent.mkdir(parents=True, exist_ok=True)+ with archive.open(item) as source, target.open('wb') as stream:+ shutil.copyfileobj(source, stream, 1024 * 1024)+ if not all(work.joinpath(*parts).is_file() for parts in MODULES):+ raise PackError('failed')+ write_licenses(root, work, wheels)+ (work / COMPLETE).write_text(json.dumps(dict(id=folder.name, bytes=total)), encoding='utf-8')+ shutil.rmtree(folder, ignore_errors=True)+ os.replace(work, folder)+ finally:+ shutil.rmtree(work, ignore_errors=True)+ for other in parent.iterdir():+ if other.is_dir() and other != folder and not other.name.startswith('.unpack-'):+ shutil.rmtree(other, ignore_errors=True)+++def write_licenses(root, unpacked, wheels):+ """The runtime's licence texts, with the pack and not with the installer: the name, version and+ licence of every wheel, then the licence files its metadata carries."""+ parts = ['Runtime libraries of the dubbing model pack. They are downloaded when the pack is installed.\n']+ for row in wheels:+ parts.append('\n%s\n%s %s, %s\n%s\n' % ('=' * 72, row['name'], row['version'], row['license'], '=' * 72))+ for info in sorted(unpacked.glob('%s-%s.dist-info' % (row['name'], row['version']))):+ for path in sorted(info.rglob('*')):+ if path.is_file() and path.suffix in ('', '.txt', '.md') and any(+ word in path.name.upper() for word in ('LICENSE', 'NOTICE', 'COPYING')):+ parts.append('\n--- %s ---\n%s\n' % (path.name, path.read_text(encoding='utf-8', errors='replace')))+ folder = base(root) / 'licenses'+ folder.mkdir(parents=True, exist_ok=True)+ (folder / LICENSES).write_text(''.join(parts), encoding='utf-8')+++def wheels_folder(root=None):+ return base(root) / 'wheels'+++def install(root=None, progress=None, check=None):+ """Download and verify the missing runtime wheels and model files, then unpack the runtime.+ Resumable; a file is kept only after its hash matched."""+ from .. import network, provision+ from ..adaln_assets import _download_plan+ from ..monitoring import save+ spec = catalog()+ progress = progress or (lambda value: None)+ check = check or (lambda: None)+ if not (spec.get('modelscope') or spec.get('huggingface')):+ raise PackError('not_published')+ wheels = wheel_rows(spec)+ if wheels is None:+ raise PackError('unsupported')+ folder = directory(root)+ folder.mkdir(parents=True, exist_ok=True)+ cache = wheels_folder(root)+ cache.mkdir(parents=True, exist_ok=True)+ ledger = _ledger(folder)+ cached = _ledger(cache)+ # A file without a stamp (a copied folder) is hashed once here and stamped.+ todo = [row for row in rows(spec) if not provision.verified(folder / row['file'], row, ledger)]+ save(folder / LEDGER, ledger)+ fetch = [] if _unpacked(root, spec) else [row for row in wheels if not provision.verified(cache / row['file'], row, cached)]+ save(cache / LEDGER, cached)+ total = sum(row['bytes'] for row in todo) + sum(row['bytes'] for row in fetch)+ retained = 0+ for row, where in [(row, folder) for row in todo] + [(row, cache) for row in fetch]:+ partial = (where / row['file']).with_suffix((Path(row['file']).suffix) + '.partial')+ if partial.is_file() and partial.stat().st_size < row['bytes']:+ retained += partial.stat().st_size+ if total and disk_space.free_bytes(folder) < total - retained + RESERVE:+ raise PackError('disk_space')+ plan = _download_plan(root)+ # The first dubbing download follows the same hosts as the image model; installs whose plan has+ # no entry for the family try ModelScope first, then the two Hugging Face hosts.+ sources = plan['sources'] = dict(plan.get('sources', {}))+ if not sources.get(FAMILY):+ sources[FAMILY] = [{'id': 'modelscope'}] + [+ dict(item) for item in sources.get('models', []) if item['id'] in ('official', 'hf-mirror')]+ if len(sources[FAMILY]) == 1:+ sources[FAMILY] += [{'id': 'official'}, {'id': 'hf-mirror'}]+ plan.update(quiet=True, resource_check=check)+ wheel_plan = runtime_plan(plan)+ done = 0++ def transfer(row, path, remote_row, ledger_file, stamps, plan=plan):+ nonlocal done+ check()+ path.parent.mkdir(parents=True, exist_ok=True)++ def moved(current, size, speed, **kwargs):+ progress(dict(phase='download', done=done + current, total=total))+ network.download(network.model_urls(plan, remote_row), path, row['sha256'], moved,+ network=plan, size=row['bytes'], category=FAMILY, keep_partial=True,+ stall_seconds=30, slow_seconds=15, low_speed_limit=64 * 1024)+ stamps[str(path)] = provision.file_identity(path, row)+ save(ledger_file, stamps)+ done += row['bytes']+ try:+ for row in fetch:+ transfer(row, cache / row['file'], runtime_remote(row), cache / LEDGER, cached, plan=wheel_plan)+ if fetch or not _unpacked(root, spec):+ progress(dict(phase='unpack', done=done, total=total))+ _unpack(root, wheels, check)+ for row in todo:+ transfer(row, folder / row['file'], remote(spec, row), folder / LEDGER, ledger)+ except PackError:+ raise+ except InterruptedError:+ raise+ except ValueError as error:+ raise PackError('disk_space' if str(error) == 'disk_space' else 'download_failed') from error+ except OSError as error:+ raise PackError('disk_space' if getattr(error, 'errno', None) == 28 else 'download_failed') from error+ except Exception as error:+ raise PackError('download_failed') from errordiff --git a/freevideo_engine/dubbing/reference.py b/freevideo_engine/dubbing/reference.pynew file mode 100644index 0000000..89ea87a--- /dev/null+++ b/freevideo_engine/dubbing/reference.py@@ -0,0 +1,39 @@+"""The voice reference of a dubbing job: an audio file, or the sound of a video, as a short mono wav."""+import numpy as np++from . import backends, wavio++RATE = 24000+MIN_SECONDS = 1.0+++def extract(source, target, start=0., seconds=backends.MAX_REFERENCE_SECONDS):+ """Decode up to `seconds` of the first audio stream of `source` (audio or video) from `start`+ into a mono 16-bit wav at `target`. Returns the seconds written."""+ import av+ chunks = []+ wanted = int(seconds * RATE)+ got = 0+ with av.open(str(source)) as container:+ if not container.streams.audio:+ raise ValueError('no_audio')+ stream = container.streams.audio[0]+ if start > 0:+ container.seek(int(start / stream.time_base) if stream.time_base else 0, stream=stream)+ resampler = av.AudioResampler(format='flt', layout='mono', rate=RATE)+ for frame in container.decode(stream):+ if frame.time is not None and frame.time + frame.samples / max(frame.rate, 1) <= start:+ continue+ for part in resampler.resample(frame):+ chunk = part.to_ndarray().reshape(-1)+ chunks.append(chunk)+ got += len(chunk)+ if got >= wanted:+ break+ if not chunks:+ raise ValueError('no_audio')+ samples = np.concatenate(chunks)[:wanted]+ if len(samples) < MIN_SECONDS * RATE:+ raise ValueError('reference_short')+ wavio.write(target, samples, RATE)+ return len(samples) / RATEdiff --git a/freevideo_engine/dubbing/service.py b/freevideo_engine/dubbing/service.pynew file mode 100644index 0000000..b4fc861--- /dev/null+++ b/freevideo_engine/dubbing/service.py@@ -0,0 +1,508 @@+"""Dubbing for the editor page, the Agent and the voice change: one job at a time, one interface.++A job takes lines (text and target seconds) and one reference voice, and writes one wav per line+into ComfyUI's input folder (`freevideo-dubbing/<job>/`), where the other tools already find media.+The model runs in its own process, so its memory leaves with it. The model is downloaded on the+first use after the user agreed (`install`).+"""+import json+import logging+import os+from pathlib import Path+import secrets+import shutil+import subprocess+import threading+import time+import uuid++from . import backends, pack+from ..prompt_vlm.service import Worker++OUTPUT = 'freevideo-dubbing'+MAX_LINES = 500+MAX_TEXT = 2000+MAX_SECONDS = 120.+MIN_SECONDS = .3+RUN_SECONDS = 300+LINE_SECONDS = 45+RUN_LIMIT = 3600+KEPT_RUNS = 8+LINE_KEYS = ('id', 'seconds', 'target', 'samples', 'onset', 'offset', 'onset_error', 'offset_error', 'error', 'speed',+ 'passes', 'stretch', 'fitted', 'warning', 'overrun', 'gain_db', 'reference', 'code')+CODES = {'invalid_request', 'unsupported_media', 'busy', 'model_missing', 'consent_required', 'unknown_job',+ 'reference_text_required', 'reference_short', 'no_audio'}+KNOWN = {'disk_space', 'download_failed', 'not_published', 'unsupported', 'timeout', 'worker_exit', 'worker_cleanup_failed',+ 'out_of_memory', 'model_missing', 'invalid_request', 'reference_text_required', 'reference_short'}+++def reference_of(value, *, required):+ """{file, text, start[, end]} of a job or line reference; None when absent and not required."""+ if value is None and not required:+ return None+ if not isinstance(value, dict) or not isinstance(value.get('file'), str) or not value['file']:+ raise ValueError('invalid_request')+ text = value.get('text', '')+ start = value.get('start', 0)+ end = value.get('end')+ if (not isinstance(text, str) or len(text) > 1000 or isinstance(start, bool)+ or not isinstance(start, (int, float)) or not 0 <= start <= 36000):+ raise ValueError('invalid_request')+ result = dict(file=value['file'], text=text.strip(), start=float(start))+ if end is not None:+ if isinstance(end, bool) or not isinstance(end, (int, float)) or not start < end <= 36000:+ raise ValueError('invalid_request')+ result['end'] = float(end)+ return result+++def request(value):+ """The job a page, the Agent or the voice change may send; anything else is refused."""+ if not isinstance(value, dict):+ raise ValueError('invalid_request')+ lines = value.get('lines')+ if not isinstance(lines, list) or not 0 < len(lines) <= MAX_LINES:+ raise ValueError('invalid_request')+ clean = []+ seen = set()+ for index, line in enumerate(lines):+ if not isinstance(line, dict):+ raise ValueError('invalid_request')+ text = line.get('text')+ seconds = line.get('seconds')+ name = str(line.get('id', 'line%03d' % index))+ if (not isinstance(text, str) or not text.strip() or len(text) > MAX_TEXT+ or isinstance(seconds, bool) or not isinstance(seconds, (int, float))+ or not MIN_SECONDS <= seconds <= MAX_SECONDS+ or not name or len(name) > 64 or not all(c.isalnum() or c in '-_' for c in name)+ or name in seen):+ raise ValueError('invalid_request')+ seen.add(name)+ row = dict(id=name, text=text.strip(), seconds=float(seconds))+ onset, offset = line.get('onset'), line.get('offset')+ for key, number in (('onset', onset), ('offset', offset)):+ if number is None:+ continue+ if isinstance(number, bool) or not isinstance(number, (int, float)) or not 0 <= number <= seconds + 1e-9:+ raise ValueError('invalid_request')+ row[key] = float(number)+ if row.get('offset', seconds) - row.get('onset', 0.) < MIN_SECONDS / 2:+ raise ValueError('invalid_request')+ if line.get('reference') is not None:+ row['reference'] = reference_of(line['reference'], required=True)+ instruction = line.get('instruction')+ if instruction not in (None, ''):+ if not isinstance(instruction, str) or len(instruction) > 200:+ raise ValueError('invalid_request')+ row['instruction'] = instruction.strip()+ clean.append(row)+ reference = reference_of(value.get('reference'), required=not all('reference' in row for row in clean))+ result = dict(lines=clean)+ if reference is not None:+ result['reference'] = reference+ seed = value.get('seed')+ if seed is not None:+ if isinstance(seed, bool) or not isinstance(seed, int) or not 0 <= seed < 2**31:+ raise ValueError('invalid_request')+ result['seed'] = seed+ name = value.get('backend')+ if name is not None:+ if name not in backends.names():+ raise ValueError('invalid_request')+ result['backend'] = name+ return result+++class Service:+ def __init__(self, installation, input_directory):+ self.installation = installation+ self.input_directory = input_directory+ self.mutex = threading.Lock()+ self.stop = threading.Event()+ self.state = dict(phase='idle')+ self.key = None+ self.thread = None+ self.process = None+ self.result_expires = 0.+ self.report = {}+ self.closed = False+ import atexit+ atexit.register(self.close)++ def info(self, wait=True):+ root, _ = self.installation()+ try:+ state = pack.status(root)+ except (OSError, ValueError):+ return dict(ready=False, bytes=0, busy=self.busy(), backends=backends.names())+ if wait and not state['ready']:+ state['ready'] = pack.settle(root)+ if state['ready']:+ state['bytes'] = 0+ return dict(ready=state['ready'], model=pack.NAME, license=state['license'], bytes=state['bytes'],+ published=state['published'], supported=state['supported'], busy=self.busy(), backends=backends.names())++ def busy(self):+ return self.thread is not None and self.thread.is_alive()++ def check(self):+ if self.stop.is_set():+ raise InterruptedError('cancelled')++ def progress(self, row):+ with self.mutex:+ self.state.update(row)++ def start(self, action, value):+ if action not in ('install', 'dub'):+ raise ValueError('invalid_request')+ if self.busy():+ raise ValueError('busy')+ root, machine = self.installation()+ if action == 'install':+ if not isinstance(value, dict) or value.get('accept_download') is not True:+ raise ValueError('consent_required')+ else:+ if not pack.settle(root):+ raise ValueError('model_missing')+ value = request(value)+ for item in [value.get('reference')] + [line.get('reference') for line in value['lines']]:+ if item:+ item['path'] = self.resolve(item['file'])+ with self.mutex:+ if self.busy():+ raise ValueError('busy')+ self.key = secrets.token_urlsafe(24)+ self.stop.clear()+ self.result_expires = 0.+ self.state = dict(phase='starting')+ self.report = {}+ self.thread = threading.Thread(target=self.work, args=(root, machine, action, value),+ name='freevideo-dubbing', daemon=True)+ self.thread.start()+ return dict(job=self.key)++ def resolve(self, name):+ """A reference inside ComfyUI's input folder; audio, or a video whose sound is used."""+ from ..comfy_assets import media_kind, resolve_assets+ resolved = resolve_assets(json.dumps([dict(file=name, role='reference')]), self.input_directory())+ path = resolved['references'][0]['path']+ if media_kind(path) not in ('audio', 'video'):+ raise ValueError('unsupported_media')+ return path++ def status(self, key):+ if not isinstance(key, str) or not self.key or not secrets.compare_digest(key, self.key):+ raise ValueError('unknown_job')+ with self.mutex:+ if self.result_expires and time.monotonic() > self.result_expires:+ self.state.pop('lines', None)+ self.state.update(phase='failed', error='expired')+ row = dict(self.state)+ # Do not announce completion while the runtime lease is still held.+ if self.busy() and row.get('phase') in ('complete', 'failed', 'cancelled'):+ row = dict(phase='finishing')+ return row++ def cancel(self, key):+ self.status(key)+ self.stop.set()+ return dict(phase='cancelling')++ def close(self):+ self.closed = True+ self.stop.set()+ process = self.process+ if process is not None and process.poll() is None:+ from .. import processes+ processes.stop(process, grace=5)++ @staticmethod+ def release_others():+ """Give the GPU back: compile, idle image worker, resident engine process, ComfyUI models."""+ from ..background_compile import COMPILE+ COMPILE.stop()+ try:+ from server import PromptServer+ from ..image_gen.service import release_idle_worker+ release_idle_worker(PromptServer.instance)+ except Exception:+ pass+ from ..resident_process import OWNER+ if OWNER.active:+ raise BlockingIOError('busy')+ OWNER.stop_prewarm()+ from ..encoder_prewarm import IDLE+ IDLE.stop()+ from ..comfy_residency import release_native+ from ..image_gen.service import comfy_server+ try:+ release_native(comfy_server())+ except Exception: # only frees memory early; the worker's own allocation still decides+ logging.warning('[FreeVideo] Could not unload ComfyUI models before dubbing', exc_info=True)++ def output_folder(self, key_id):+ base = Path(self.input_directory()) / OUTPUT+ base.mkdir(parents=True, exist_ok=True)+ # Keep the latest few runs; older ones are the page's old results.+ runs = sorted((p for p in base.iterdir() if p.is_dir()), key=lambda p: p.stat().st_mtime)+ for old in runs[:max(0, len(runs) - KEPT_RUNS + 1)]:+ shutil.rmtree(old, ignore_errors=True)+ folder = base / key_id+ folder.mkdir()+ return folder++ def work(self, root, machine, action, value):+ from ..locking import runtime_lock+ from ..monitoring import save+ started = time.monotonic()+ try:+ if action == 'install':+ with runtime_lock(Path(root) / 'dubbing-install.lock', inherit=False):+ self.check()+ pack.install(root, self.progress, self.check)+ self.check()+ self.progress(dict(phase='complete'))+ else:+ self.release_others()+ with runtime_lock(Path(root) / 'engine.lock', inherit=False) as descriptor:+ self.check()+ self.dub(root, machine, descriptor, value)+ except InterruptedError:+ self.progress(dict(phase='cancelled'))+ except BlockingIOError:+ self.progress(dict(phase='failed', error='busy'))+ except pack.PackError as error:+ self.progress(dict(phase='failed', error=error.reason))+ except Exception as error:+ reason = str(error)+ self.progress(dict(phase='failed', exception=type(error).__name__,+ error=reason if reason in KNOWN else 'download_failed' if action == 'install'+ else 'dub_failed'))+ finally:+ if self.stop.is_set():+ self.progress(dict(phase='cancelled'))+ self.result_expires = time.monotonic() + 600+ # Allowlisted metadata only: never the text, file names or paths.+ report = {key: item for key, item in self.state.items()+ if key in ('phase', 'error', 'exception', 'timings', 'worker_returncode', 'backend',+ 'sample_rate', 'reference_seconds', 'loaded_seconds')}+ if self.state.get('lines'):+ report['lines'] = [{key: row.get(key) for key in LINE_KEYS if key not in ('id', 'reference')}+ for row in self.state['lines']]+ report.update(schema='freevideo.dubbing', schema_version=1, action=action, created_epoch=time.time(),+ report_id=uuid.uuid4().hex, elapsed_seconds=time.monotonic() - started)+ self.report = report+ try:+ save(Path(root) / 'dubbing-report.json', report)+ except OSError:+ pass++ @staticmethod+ def environment(root):+ from ..comfy_environment import isolated_environment+ from ..comfy_bridge import source_root+ from .. import processes, resident_process+ env = isolated_environment(root, source_root())+ for name in ('FREEVIDEO_RUNTIME_LOCK_FD', processes.HANDLE_ENV, resident_process.ENV):+ env.pop(name, None)+ return env++ @staticmethod+ def spawn(root, machine, env, descriptor=None):+ from .. import processes+ command = processes.module_command('freevideo_engine.dubbing.worker', pack.directory(root), pack.runtime_path(root) or '')+ command[0] = machine['comfy_python']+ return processes.popen(command, env=env, stdin=subprocess.PIPE, stdout=subprocess.PIPE,+ stderr=subprocess.PIPE, text=True, encoding='utf-8', errors='replace',+ pass_fds=() if descriptor is None else (descriptor,),+ supervise=True, start_new_session=True)++ def dub(self, root, machine, descriptor, value):+ from .. import processes, resident_process+ from . import reference+ import psutil+ env = self.environment(root)+ key_id = uuid.uuid4().hex[:12]+ folder = self.output_folder(key_id)+ self.progress(dict(phase='preparing'))+ extracted = {}++ def extract(ref, name):+ key = (ref['path'], ref['start'], ref.get('end'))+ if key not in extracted:+ wav = folder / name+ span = ref['end'] - ref['start'] if ref.get('end') else backends.MAX_REFERENCE_SECONDS+ try:+ reference.extract(ref['path'], wav, start=ref['start'], seconds=min(span, backends.MAX_REFERENCE_SECONDS))+ except ValueError as error:+ if str(error) not in CODES:+ raise ValueError('unsupported_media') from None+ extracted[key] = str(error)+ return extracted[key]+ except Exception:+ raise ValueError('unsupported_media') from None+ extracted[key] = str(wav)+ return extracted[key]+ job = dict(value, output_dir=str(folder))+ if value.get('reference'):+ made = extract(value['reference'], 'reference.wav')+ if made not in (str(folder / 'reference.wav'),):+ raise ValueError(made)+ job['reference'] = dict(path=made, text=value['reference']['text'])+ job['lines'] = []+ for index, line in enumerate(value['lines']):+ row = {key: item for key, item in line.items() if key != 'reference'}+ if line.get('reference'):+ made = extract(line['reference'], 'reference-%d.wav' % index)+ if os.path.isfile(made):+ row['reference'] = dict(path=made, text=line['reference']['text'])+ elif made == 'reference_short' and job.get('reference'):+ # the job's reference is used, and the line says so+ row['reference_short'] = True+ else:+ raise ValueError(made)+ job['lines'].append(row)+ session = Worker(self.spawn(root, machine, env, descriptor))+ child = session.process+ self.process = child+ result = {}+ lines = []+ write_errors = []++ def submit():+ try:+ child.stdin.write(json.dumps(job, ensure_ascii=True))+ child.stdin.close()+ except Exception as error:+ write_errors.append(error)+ writer = threading.Thread(target=submit, daemon=True)++ def read_rows():+ import queue+ while True:+ try:+ row = session.rows.get_nowait()+ except queue.Empty:+ return+ phase = row.get('phase')+ if phase in ('complete', 'failed', 'cancelled'):+ result.update(row)+ elif phase == 'line':+ entry = {key: row[key] for key in LINE_KEYS if key in row}+ if row.get('file'):+ entry['file'] = '%s/%s/%s' % (OUTPUT, key_id, row['file'])+ lines.append(entry)+ self.progress(dict(phase='synthesizing', index=row.get('index'), count=len(job['lines']),+ lines=list(lines)))+ elif phase in ('loading', 'loaded', 'synthesizing'):+ self.progress(dict((k, v) for k, v in row.items() if k in ('phase', 'index', 'count', 'seconds')))+ observed = {}+ try:+ writer.start()+ deadline = time.monotonic() + min(RUN_LIMIT, RUN_SECONDS + LINE_SECONDS * len(job['lines']))+ while child.poll() is None:+ read_rows()+ self.check()+ if write_errors:+ raise write_errors[0]+ if time.monotonic() > deadline:+ raise TimeoutError('timeout')+ try:+ for process in psutil.Process(child.pid).children(recursive=True):+ observed[process.pid] = process+ except psutil.Error:+ pass+ self.stop.wait(.25)+ session.reader.join(timeout=5)+ read_rows()+ self.check()+ if session.reader.is_alive() or not result:+ raise RuntimeError('worker_exit')+ if result.get('phase') == 'failed':+ raise RuntimeError(result.get('code') if result.get('code') in KNOWN else 'dub_failed')+ if result.get('phase') == 'cancelled':+ raise InterruptedError('cancelled')+ if child.returncode:+ raise RuntimeError('worker_exit')+ self.progress(dict(phase='complete', lines=lines, backend=result.get('backend'),+ sample_rate=result.get('sample_rate'),+ reference_seconds=result.get('reference_seconds'),+ worker_returncode=child.returncode))+ finally:+ if child.poll() is None:+ processes.stop(child, grace=15)+ child.wait()+ writer.join(timeout=5)+ session.close()+ self.process = None+ for wav in folder.glob('reference*.wav'):+ try:+ wav.unlink()+ except OSError:+ pass+ surviving = []+ for process in observed.values():+ try:+ if process.is_running() and process.status() != 'zombie':+ surviving.append(process)+ except psutil.NoSuchProcess:+ pass+ for process in surviving:+ try:+ process.kill()+ except psutil.NoSuchProcess:+ pass+ if surviving:+ psutil.wait_procs(surviving, timeout=5)+ self.progress(dict(error='worker_cleanup_failed'))+ # Failed or cancelled runs leave nothing behind.+ if self.state.get('phase') != 'complete':+ shutil.rmtree(folder, ignore_errors=True)+++def register():+ import asyncio+ from aiohttp import web+ from server import PromptServer+ import folder_paths+ from ..comfy_bridge import installation+ server = PromptServer.instance+ if server is None or getattr(server, '_freevideo_dubbing', None) is not None:+ return+ service = Service(installation, folder_paths.get_input_directory)+ server._freevideo_dubbing = service+ token = secrets.token_urlsafe(24)++ @server.routes.get('/freevideo/dubbing')+ async def info(request):+ try:+ return web.json_response(dict(await asyncio.to_thread(service.info, False), token=token))+ except (OSError, ValueError):+ return web.json_response(dict(error='setup_required'), status=400)++ @server.routes.post('/freevideo/dubbing/{action}')+ async def action(request):+ if not secrets.compare_digest(request.headers.get('X-FreeVideo-Dubbing', ''), token):+ return web.json_response(dict(error='reload'), status=403)+ try:+ name = request.match_info['action']+ if request.content_length and request.content_length > 512 * 1024:+ raise ValueError('invalid_request')+ value = await request.json()+ if not isinstance(value, dict):+ raise ValueError('invalid_request')+ if name == 'status':+ result = service.status(value.get('job'))+ elif name == 'cancel':+ result = service.cancel(value.get('job'))+ else:+ running, pending = server.prompt_queue.get_current_queue()+ if running or pending:+ raise ValueError('busy')+ if name == 'dub':+ await asyncio.to_thread(pack.settle, service.installation()[0])+ result = service.start(name, value)+ return web.json_response(result)+ except (ValueError, OSError) as error:+ return web.json_response(dict(error=str(error) if str(error) in CODES else 'invalid_media'), status=400)diff --git a/freevideo_engine/dubbing/wavio.py b/freevideo_engine/dubbing/wavio.pynew file mode 100644index 0000000..9112b24--- /dev/null+++ b/freevideo_engine/dubbing/wavio.py@@ -0,0 +1,27 @@+"""16-bit PCM wav files, read and written without an audio library."""+import wave++import numpy as np+++def read(path):+ """Float32 samples shaped (frames, channels) and the sample rate of a 16-bit PCM wav."""+ with wave.open(str(path), 'rb') as file:+ if file.getsampwidth() != 2:+ raise ValueError('unsupported_wav')+ channels, rate = file.getnchannels(), file.getframerate()+ raw = file.readframes(file.getnframes())+ data = np.frombuffer(raw, dtype='<i2').astype(np.float32) / 32768.+ return data.reshape(-1, channels), rate+++def write(path, samples, rate):+ """Write mono (n,) or (n, channels) float samples in [-1, 1] as 16-bit PCM."""+ data = np.asarray(samples, dtype=np.float32)+ channels = 1 if data.ndim == 1 else data.shape[1]+ pcm = np.round(np.clip(data, -1., 1.) * 32767.).astype('<i2')+ with wave.open(str(path), 'wb') as file:+ file.setnchannels(channels)+ file.setsampwidth(2)+ file.setframerate(int(rate))+ file.writeframes(pcm.tobytes())diff --git a/freevideo_engine/dubbing/worker.py b/freevideo_engine/dubbing/worker.pynew file mode 100644index 0000000..8bca359--- /dev/null+++ b/freevideo_engine/dubbing/worker.py@@ -0,0 +1,297 @@+"""One dubbing job per process: the model, its CUDA context and its memory leave with the process.++Arguments: the model folder, then the folder of the downloaded runtime libraries (optional).+The request is one JSON object on stdin; progress and the result are JSON lines on stdout. A line is+fitted to its target duration by `align.fit`, then written as a wav file in `output_dir`.+"""+import json+import math+import os+import sys+import time+import traceback++from . import align, backends++IDLE_SECONDS = 600+ROW_LIMIT = 128 * 1024+++def emit(**row):+ sys.stdout.write(json.dumps(row, ensure_ascii=False, separators=(',', ':')) + '\n')+ sys.stdout.flush()+++LEVEL_TARGET = 0.06+LEVEL_PEAK = 0.95+LEVEL_GAIN = 10.+++def leveled(data, rate=None):+ """The model gives back the level of its reference, so a quiet recording would come back nearly+ silent. Samples quieter than LEVEL_TARGET (RMS) are raised towards it: the gain is 1 at the target and+ grows smoothly as the recording gets quieter, never below 1, never past LEVEL_GAIN, never into clipping.+ With `rate` the level is that of the voiced part, so the silence around a line cut from a separated+ voice does not raise the line more than its voice needs; a recording that is loud as a whole is left+ as it is."""+ import numpy as np+ rms = float(np.sqrt(np.mean(np.square(data, dtype=np.float64)))) if len(data) else 0.+ peak = float(np.max(np.abs(data))) if len(data) else 0.+ if rms <= 1e-5 or rms >= LEVEL_TARGET:+ return data, 1.+ if rate:+ rms = voiced_level(data, rate) or rms+ gain = max(1., min(LEVEL_TARGET / rms, LEVEL_PEAK / peak, LEVEL_GAIN))+ if gain == 1.:+ return data, 1.+ return data * np.float32(gain), gain+++MATCH_RANGE = (0.1, 10.)+++def voiced_level(data, rate):+ """RMS of the voiced part of `data` (None when there is none)."""+ import numpy as np+ span = align.voiced_span(data, rate)+ if span is None:+ return None+ part = np.asarray(data[span[0]:span[1]], dtype=np.float64)+ return float(np.sqrt(np.mean(np.square(part)))) if len(part) else None+++def matched(samples, rate, onset, offset, level):+ """`samples` scaled so that the voiced part (from `onset` to `offset` seconds) has RMS `level`: the line+ comes back as loud as the original line it replaces, whatever level the model was given. Returns+ (samples, gain); the gain stays inside MATCH_RANGE and never clips."""+ import numpy as np+ current = voiced_level(samples[int(onset * rate):int(offset * rate)], rate)+ peak = float(np.max(np.abs(samples))) if len(samples) else 0.+ if not level or not current or peak <= 0:+ return samples, 1.+ gain = min(max(level / current, MATCH_RANGE[0]), MATCH_RANGE[1], LEVEL_PEAK / peak)+ if abs(gain - 1.) < 0.01:+ return samples, 1.+ return samples * np.float32(gain), gain+++CLAUSE_END = '。!?;,、.!?;,\n'+++def cut_text(text, share):+ """The longest start of `text` that is no more than `share` of it and ends where a clause or a word+ ends; returns (start of the text, its share of the whole text)."""+ limit = int(len(text) * share)+ if limit <= 0:+ return '', 0.+ stops = [i for i in range(limit, 0, -1) if text[i - 1] in CLAUSE_END or text[i - 1] == ' ' or text[i:i + 1] == ' ']+ end = next((i for i in stops if i >= limit // 2), limit)+ kept = text[:end].strip()+ return kept, len(kept) / len(text)+++def usable_reference(path, folder, name='reference-prepared.wav', text='', texts=None):+ """Return a mono wav no longer than the models accept and not too quiet (a changed copy if either+ applies), and its seconds. A reference that is too long is cut where its text is cut (the model reads the+ text it is given, so a text longer than the audio is spoken into the line): the cut text goes to+ `texts[name]`."""+ from . import wavio+ import numpy as np+ data, rate = wavio.read(path)+ seconds = len(data) / rate+ if seconds < 1.0:+ raise ValueError('reference_short')+ changed = False+ if texts is not None:+ texts[name] = text+ if seconds > backends.MAX_REFERENCE_SECONDS:+ end = int(backends.MAX_REFERENCE_SECONDS * rate)+ if text:+ kept, share = cut_text(text, backends.MAX_REFERENCE_SECONDS / seconds)+ if kept:+ text, end = kept, int(len(data) * share)+ # the cut falls on the quietest 10 ms of the last half second+ frame = max(1, rate // 100)+ first = max(0, end - rate // 2) // frame+ count = (end - first * frame) // frame+ if count > 1:+ rms = np.sqrt(np.square(np.asarray(data[first * frame:(first + count) * frame], dtype=np.float64))+ .reshape(count, frame).mean(axis=1))+ end = (first + int(np.argmin(rms))) * frame + frame // 2+ if texts is not None:+ texts[name] = text+ data = data[:end]+ seconds = len(data) / rate+ changed = True+ data, gain = leveled(data, rate)+ if gain != 1.:+ changed = True+ if not changed:+ return path, seconds+ target = os.path.join(folder, name)+ wavio.write(target, data, rate)+ return target, seconds+++def validate(request):+ lines = request.get('lines')+ if not isinstance(lines, list) or not lines or len(lines) > 500:+ raise ValueError('invalid_request')+ for line in lines:+ if not isinstance(line, dict) or not str(line.get('text') or '').strip():+ raise ValueError('invalid_request')+ if not float(line.get('seconds') or 0) > 0:+ raise ValueError('invalid_request')+ reference = request.get('reference') or {}+ own = [(line.get('reference') or {}) for line in lines]+ if any(not isinstance(item, dict) for item in own):+ raise ValueError('invalid_request')+ for item in own:+ if item and not os.path.isfile(str(item.get('path') or '')):+ raise ValueError('invalid_request')+ if (reference or not all(own)) and not os.path.isfile(str(reference.get('path') or '')):+ raise ValueError('invalid_request')+ if not os.path.isdir(str(request.get('output_dir') or '')):+ raise ValueError('invalid_request')+ return lines, reference+++def run(request, model_dir, device='cuda'):+ from . import wavio+ from ..prompt_vlm.worker import Cancelled+ lines, reference = validate(request)+ folder = request['output_dir']+ name = request.get('backend') or backends.DEFAULT+ seed = request.get('seed')+ # References are prepared before the model loads: a bad one is refused in seconds, not after the load.+ prepared = {}+ levels = {}+ cut = {}++ def prepare(source, key):+ if key not in prepared:+ name = 'reference-%s.wav' % key+ prepared[key] = usable_reference(source['path'], folder, name, str(source.get('text') or '').strip(), cut)+ data, rate = wavio.read(source['path'])+ levels[key] = voiced_level(data, rate)+ return prepared[key]+ job_ref = None+ if reference:+ job_ref = prepare(reference, 'job') + (cut['reference-job.wav'],)+ used = []+ match = {} # per line with its own reference: the voiced RMS the line is to come back with+ for index, line in enumerate(lines):+ own = line.get('reference')+ if own:+ try:+ used.append(('line',) + prepare(own, 'line%d' % index) + (cut['reference-line%d.wav' % index], None))+ match[index] = levels.get('line%d' % index)+ continue+ except ValueError as error:+ if str(error) != 'reference_short' or job_ref is None:+ raise+ used.append(('job',) + job_ref + ('reference_short',))+ continue+ used.append(('job',) + job_ref + ('reference_short' if line.get('reference_short') else None,))+ emit(phase='loading')+ started = time.monotonic()+ backend = backends.create(name, model_dir, device=device)+ emit(phase='loaded', seconds=round(time.monotonic() - started, 2))+ try:+ if backend.needs_reference_text and any(not item[3] for item in used):+ raise ValueError('reference_text_required')+ results = []+ for index, line in enumerate(lines):+ emit(phase='synthesizing', index=index, count=len(lines))+ source, ref_path, ref_seconds, ref_text, note = used[index]+ if seed is not None:+ from .cosyvoice3.utils.common import set_all_random_seed+ set_all_random_seed(int(seed) + index)+ instruction = (line.get('instruction') or '').strip() or None+ text = str(line['text']).strip()+ target = float(line['seconds'])+ row = dict(id=line.get('id'), target=target, reference=source)+ try:+ def synth(speed, text=text, instruction=instruction, ref_path=ref_path, ref_text=ref_text):+ return backend.synth(text, ref_path, ref_text, speed, instruction)++ fitted = align.fit(synth, target, onset=float(line.get('onset') or 0.),+ offset=None if line.get('offset') is None else float(line['offset']))+ except Cancelled:+ raise+ except Exception as error:+ traceback.print_exc(file=sys.stderr)+ row.update(failure(error))+ results.append(row)+ emit(phase='line', index=index, **row)+ continue+ samples, gain = fitted['samples'], 1.+ if match.get(index):+ samples, gain = matched(samples, fitted['rate'], fitted['onset'], fitted['offset'], match[index])+ path = os.path.join(folder, '%s.wav' % (line.get('id') or 'line%03d' % index))+ wavio.write(path, samples, fitted['rate'])+ row.update(file=os.path.basename(path), seconds=round(fitted['seconds'], 4), samples=len(fitted['samples']),+ onset=round(fitted['onset'], 3), offset=round(fitted['offset'], 3),+ onset_error=round(fitted['onset_error'], 3), offset_error=round(fitted['offset_error'], 3),+ error=round(fitted['error'], 3), speed=round(fitted['speed'], 3), passes=fitted['passes'],+ stretch=round(fitted['stretch'], 3), fitted=bool(fitted['fitted']),+ warning=fitted['warning'] or (note if note == 'reference_short' else None),+ overrun=round(fitted['overrun'], 3), gain_db=round(20 * math.log10(gain), 1))+ results.append(row)+ emit(phase='line', index=index, **row)+ finally:+ backend.close()+ if all('code' in row for row in results):+ raise _LineFailure(results[0])+ return dict(lines=results, reference_seconds=round(used[0][2], 2), backend=name, sample_rate=backend.sample_rate)+++class _LineFailure(Exception):+ """Every line failed: the job fails with the first line's code."""+ def __init__(self, row):+ super().__init__(row['code'])+ self.row = row+++CODES = ('invalid_request', 'reference_text_required', 'reference_short')+++def failure(error):+ if isinstance(error, _LineFailure):+ return dict(code=error.row['code'], **({'error': error.row['error']} if 'error' in error.row else {}))+ text = str(error)+ if isinstance(error, ValueError) and text in CODES:+ return dict(code=text)+ if isinstance(error, FileNotFoundError):+ return dict(code='model_missing')+ low = text.lower()+ if isinstance(error, MemoryError) or 'out of memory' in low:+ return dict(code='out_of_memory')+ return dict(code='failed', error=type(error).__name__)+++def main():+ from ..prompt_vlm.worker import Cancelled, stop_on_signal+ stop_on_signal()+ try:+ request = json.loads(sys.stdin.buffer.read(1024 * 1024))+ if not isinstance(request, dict):+ raise ValueError('invalid_request')+ if len(sys.argv) > 2 and sys.argv[2]:+ # -I ignores PYTHONPATH: the downloaded runtime (onnxruntime) is handed over as an argument.+ from .pack import activate+ activate(sys.argv[2])+ result = run(request, sys.argv[1], device=request.get('device') or 'cuda')+ emit(phase='complete', **result)+ return 0+ except Cancelled:+ emit(phase='cancelled')+ return 0+ except Exception as error:+ traceback.print_exc(file=sys.stderr)+ emit(phase='failed', **failure(error))+ return 1+++if __name__ == '__main__':+ raise SystemExit(main())diff --git a/freevideo_engine/licenses/CosyVoice-Apache-2.0.txt b/freevideo_engine/licenses/CosyVoice-Apache-2.0.txtnew file mode 100644index 0000000..31e1c88--- /dev/null+++ b/freevideo_engine/licenses/CosyVoice-Apache-2.0.txt@@ -0,0 +1,206 @@+Copyright (c) 2024 Alibaba Inc (FunAudioLLM). CosyVoice, Fun-CosyVoice3-0.5B-2512 and the code in+freevideo_engine/dubbing/cosyvoice3 adapted from https://github.com/FunAudioLLM/CosyVoice are licensed+under the Apache License 2.0 reproduced below. FreeVideo reduced the code to the zero-shot and instruct2+inputs and changed its file, audio and optional-dependency handling; each source file keeps its header.++ Apache License+ Version 2.0, January 2004+ http://www.apache.org/licenses/++ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION++ 1. Definitions.++ "License" shall mean the terms and conditions for use, reproduction,+ and distribution as defined by Sections 1 through 9 of this document.++ "Licensor" shall mean the copyright owner or entity authorized by+ the copyright owner that is granting the License.++ "Legal Entity" shall mean the union of the acting entity and all+ other entities that control, are controlled by, or are under common+ control with that entity. For the purposes of this definition,+ "control" means (i) the power, direct or indirect, to cause the+ direction or management of such entity, whether by contract or+ otherwise, or (ii) ownership of fifty percent (50%) or more of the+ outstanding shares, or (iii) beneficial ownership of such entity.++ "You" (or "Your") shall mean an individual or Legal Entity+ exercising permissions granted by this License.++ "Source" form shall mean the preferred form for making modifications,+ including but not limited to software source code, documentation+ source, and configuration files.++ "Object" form shall mean any form resulting from mechanical+ transformation or translation of a Source form, including but+ not limited to compiled object code, generated documentation,+ and conversions to other media types.++ "Work" shall mean the work of authorship, whether in Source or+ Object form, made available under the License, as indicated by a+ copyright notice that is included in or attached to the work+ (an example is provided in the Appendix below).++ "Derivative Works" shall mean any work, whether in Source or Object+ form, that is based on (or derived from) the Work and for which the+ editorial revisions, annotations, elaborations, or other modifications+ represent, as a whole, an original work of authorship. For the purposes+ of this License, Derivative Works shall not include works that remain+ separable from, or merely link (or bind by name) to the interfaces of,+ the Work and Derivative Works thereof.++ "Contribution" shall mean any work of authorship, including+ the original version of the Work and any modifications or additions+ to that Work or Derivative Works thereof, that is intentionally+ submitted to Licensor for inclusion in the Work by the copyright owner+ or by an individual or Legal Entity authorized to submit on behalf of+ the copyright owner. For the purposes of this definition, "submitted"+ means any form of electronic, verbal, or written communication sent+ to the Licensor or its representatives, including but not limited to+ communication on electronic mailing lists, source code control systems,+ and issue tracking systems that are managed by, or on behalf of, the+ Licensor for the purpose of discussing and improving the Work, but+ excluding communication that is conspicuously marked or otherwise+ designated in writing by the copyright owner as "Not a Contribution."++ "Contributor" shall mean Licensor and any individual or Legal Entity+ on behalf of whom a Contribution has been received by Licensor and+ subsequently incorporated within the Work.++ 2. Grant of Copyright License. Subject to the terms and conditions of+ this License, each Contributor hereby grants to You a perpetual,+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable+ copyright license to reproduce, prepare Derivative Works of,+ publicly display, publicly perform, sublicense, and distribute the+ Work and such Derivative Works in Source or Object form.++ 3. Grant of Patent License. Subject to the terms and conditions of+ this License, each Contributor hereby grants to You a perpetual,+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable+ (except as stated in this section) patent license to make, have made,+ use, offer to sell, sell, import, and otherwise transfer the Work,+ where such license applies only to those patent claims licensable+ by such Contributor that are necessarily infringed by their+ Contribution(s) alone or by combination of their Contribution(s)+ with the Work to which such Contribution(s) was submitted. If You+ institute patent litigation against any entity (including a+ cross-claim or counterclaim in a lawsuit) alleging that the Work+ or a Contribution incorporated within the Work constitutes direct+ or contributory patent infringement, then any patent licenses+ granted to You under this License for that Work shall terminate+ as of the date such litigation is filed.++ 4. Redistribution. You may reproduce and distribute copies of the+ Work or Derivative Works thereof in any medium, with or without+ modifications, and in Source or Object form, provided that You+ meet the following conditions:++ (a) You must give any other recipients of the Work or+ Derivative Works a copy of this License; and++ (b) You must cause any modified files to carry prominent notices+ stating that You changed the files; and++ (c) You must retain, in the Source form of any Derivative Works+ that You distribute, all copyright, patent, trademark, and+ attribution notices from the Source form of the Work,+ excluding those notices that do not pertain to any part of+ the Derivative Works; and++ (d) If the Work includes a "NOTICE" text file as part of its+ distribution, then any Derivative Works that You distribute must+ include a readable copy of the attribution notices contained+ within such NOTICE file, excluding those notices that do not+ pertain to any part of the Derivative Works, in at least one+ of the following places: within a NOTICE text file distributed+ as part of the Derivative Works; within the Source form or+ documentation, if provided along with the Derivative Works; or,+ within a display generated by the Derivative Works, if and+ wherever such third-party notices normally appear. The contents+ of the NOTICE file are for informational purposes only and+ do not modify the License. You may add Your own attribution+ notices within Derivative Works that You distribute, alongside+ or as an addendum to the NOTICE text from the Work, provided+ that such additional attribution notices cannot be construed+ as modifying the License.++ You may add Your own copyright statement to Your modifications and+ may provide additional or different license terms and conditions+ for use, reproduction, or distribution of Your modifications, or+ for any such Derivative Works as a whole, provided Your use,+ reproduction, and distribution of the Work otherwise complies with+ the conditions stated in this License.++ 5. Submission of Contributions. Unless You explicitly state otherwise,+ any Contribution intentionally submitted for inclusion in the Work+ by You to the Licensor shall be under the terms and conditions of+ this License, without any additional terms or conditions.+ Notwithstanding the above, nothing herein shall supersede or modify+ the terms of any separate license agreement you may have executed+ with Licensor regarding such Contributions.++ 6. Trademarks. This License does not grant permission to use the trade+ names, trademarks, service marks, or product names of the Licensor,+ except as required for reasonable and customary use in describing the+ origin of the Work and reproducing the content of the NOTICE file.++ 7. Disclaimer of Warranty. Unless required by applicable law or+ agreed to in writing, Licensor provides the Work (and each+ Contributor provides its Contributions) on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or+ implied, including, without limitation, any warranties or conditions+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A+ PARTICULAR PURPOSE. You are solely responsible for determining the+ appropriateness of using or redistributing the Work and assume any+ risks associated with Your exercise of permissions under this License.++ 8. Limitation of Liability. In no event and under no legal theory,+ whether in tort (including negligence), contract, or otherwise,+ unless required by applicable law (such as deliberate and grossly+ negligent acts) or agreed to in writing, shall any Contributor be+ liable to You for damages, including any direct, indirect, special,+ incidental, or consequential damages of any character arising as a+ result of this License or out of the use or inability to use the+ Work (including but not limited to damages for loss of goodwill,+ work stoppage, computer failure or malfunction, or any and all+ other commercial damages or losses), even if such Contributor+ has been advised of the possibility of such damages.++ 9. Accepting Warranty or Additional Liability. While redistributing+ the Work or Derivative Works thereof, You may choose to offer,+ and charge a fee for, acceptance of support, warranty, indemnity,+ or other liability obligations and/or rights consistent with this+ License. However, in accepting such obligations, You may act only+ on Your own behalf and on Your sole responsibility, not on behalf+ of any other Contributor, and only if You agree to indemnify,+ defend, and hold each Contributor harmless for any liability+ incurred by, or claims asserted against, such Contributor by reason+ of your accepting any such warranty or additional liability.++ END OF TERMS AND CONDITIONS++ APPENDIX: How to apply the Apache License to your work.++ To apply the Apache License to your work, attach the following+ boilerplate notice, with the fields enclosed by brackets "[]"+ replaced with your own identifying information. (Don't include+ the brackets!) The text should be enclosed in the appropriate+ comment syntax for the file format. We also recommend that a+ file or class name and description of purpose be included on the+ same "printed page" as the copyright notice for easier+ identification within third-party archives.++ Copyright 2026 FreeToken Authors++ Licensed under the Apache License, Version 2.0 (the "License");+ you may not use this file except in compliance with the License.+ You may obtain a copy of the License at++ http://www.apache.org/licenses/LICENSE-2.0++ Unless required by applicable law or agreed to in writing, software+ distributed under the License is distributed on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ See the License for the specific language governing permissions and+ limitations under the License.diff --git a/freevideo_engine/licenses/CosyVoice-NOTICE.txt b/freevideo_engine/licenses/CosyVoice-NOTICE.txtnew file mode 100644index 0000000..482132a--- /dev/null+++ b/freevideo_engine/licenses/CosyVoice-NOTICE.txt@@ -0,0 +1,29 @@+Notices for code adapted into freevideo_engine/dubbing/cosyvoice3 (dubbing, optional download of the model).++CosyVoice / Fun-CosyVoice3-0.5B-2512, Copyright (c) 2024 Alibaba Inc (FunAudioLLM): Apache License 2.0,+see CosyVoice-Apache-2.0.txt. The model weights are downloaded on request and are not part of the installer.++The following parts keep the MIT licence of their authors:+- utils/mel.py mel_spectrogram, from Matcha-TTS (matcha/utils/audio.py), Copyright (c) 2023 Shivam Mehta,+ derived from HiFi-GAN, Copyright (c) 2020 Jungil Kong.+- flow/flow_matching.py attributes inlined from Matcha-TTS, Copyright (c) 2023 Shivam Mehta.+- flow/DiT/modules.py rotary embedding, re-implemented from x_transformers and rotary-embedding-torch,+ Copyright (c) 2020 Phil Wang.+- transformer/activation.py Snake activation, from https://github.com/EdwardDixon/snake.++MIT License++Permission is hereby granted, free of charge, to any person obtaining a copy of this software and+associated documentation files (the "Software"), to deal in the Software without restriction, including+without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the+following conditions:++The above copyright notice and this permission notice shall be included in all copies or substantial+portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT+LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO+EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE+USE OR OTHER DEALINGS IN THE SOFTWARE.diff --git a/pyproject.toml b/pyproject.tomlindex 448e585..51ea79d 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -35,5 +35,6 @@ include = ["freevideo_engine*"][tool.setuptools.package-data]"freevideo_engine.prompt_vlm" = ["model.json"]+"freevideo_engine.dubbing" = ["cosyvoice3_pack.json"]"freevideo_engine.image_gen" = ["image_models.json", "licenses/NOTICE.txt"]freevideo_engine = ["stage_estimate_params.json", "build-version.txt", "build-identity.json", "release_notes.json", "dependencies.json", "model_files.json", "model_shares.json", "prepared_models.json", "macos_reference_catalog.json", "superseded_tables.json", "bootstrap_versions.json", "curl_windows.json", "git_linux.json", "git_macos.json", "gcc_linux.json", "uv_downloads.json", "torch_downloads.json", "test_prompts.json", "windows_bootstrap.ps1", "assets/*.png", "assets/*.ico", "launcher/*.qml", "launcher/licenses/*.txt", "licenses/*.txt", "licenses/runtime/*.txt", "subtitle_pack.json", "frame_preview_weights.json", "agent/skill/freevideo/*.txt", "agent/skill/freevideo/references/*.txt", "depth_reference_model.json", "swap_lora.json"]