Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.
Subtitles: multilingual speech model in the download list, notices
Bowen Xue482308c3d4b1
diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.mdindex ed8f93a..38c8576 100644--- a/THIRD_PARTY_NOTICES.md+++ b/THIRD_PARTY_NOTICES.md@@ -19,6 +19,7 @@ FreeVideo code uses [Apache-2.0](LICENSE). Its dependencies and models retain th| [Silero VAD](https://github.com/snakers4/silero-vad) v5.0 voice-activity model, © 2020-present Silero Team (optional download; license file downloaded with the model) | [MIT](freevideo_engine/licenses/Silero-VAD-MIT.txt) || [NVIDIA NeMo stt_en_conformer_ctc_small](https://huggingface.co/nvidia/stt_en_conformer_ctc_small), © NVIDIA, exported to ONNX and quantized to int8 by the sherpa-onnx project, changed from the original; not endorsed by NVIDIA (optional download for English subtitles; [notice](freevideo_engine/licenses/NeMo-stt_en_conformer_ctc_small-NOTICE.txt)) | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) || [OpenAI Whisper tiny](https://github.com/openai/whisper), © 2022 OpenAI, exported to ONNX and quantized to int8 by the sherpa-onnx project (optional download for spoken-language detection) | [MIT](freevideo_engine/licenses/Whisper-MIT.txt) |+| [SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) by FunASR / FunAudioLLM (Alibaba Group), converted to ONNX and quantized to int8 by the sherpa-onnx project, used unchanged under its original name (optional download for multilingual subtitles; [notice](freevideo_engine/licenses/SenseVoice-NOTICE.txt)) | [FunASR Model Open Source License Agreement v1.1](freevideo_engine/licenses/FunASR-MODEL_LICENSE-v1.1.txt) || [Lucide](https://lucide.dev) icons (image editor: eyedropper, hand, crop) | [ISC](freevideo_engine/licenses/Lucide-ISC.txt) || [Noto Sans CJK SC and Noto Sans Mono CJK SC](https://github.com/notofonts/noto-cjk) 2.004, © 2014-2021 Adobe: subsets renamed FreeVideo Sans SC and FreeVideo Mono SC (image editor text, editor text and subtitles / 剪辑视图的文字和字幕, share card, Linux launcher) | [SIL OFL 1.1](freevideo_engine/licenses/Noto-CJK-Sans-OFL.txt) || [Noto Serif CJK SC](https://github.com/notofonts/noto-cjk) 2.003, © 2017-2024 Adobe: subset renamed FreeVideo Serif SC (image editor text, editor text and subtitles / 剪辑视图的文字和字幕) | [SIL OFL 1.1](freevideo_engine/licenses/Noto-CJK-Serif-OFL.txt) |diff --git a/freevideo_engine/licenses/FunASR-MODEL_LICENSE-v1.1.txt b/freevideo_engine/licenses/FunASR-MODEL_LICENSE-v1.1.txtnew file mode 100644index 0000000..d5cc52a--- /dev/null+++ b/freevideo_engine/licenses/FunASR-MODEL_LICENSE-v1.1.txt@@ -0,0 +1,107 @@+FunASR Model Open Source License Agreement++Version: 1.1++Copyright (C) [2023-2028] [Alibaba Group]. All rights reserved.++Thank you for choosing the FunASR open-source model. The FunASR open-source model includes a range of free and open industrial models for you to use, modify, share, and learn from.++To ensure better community collaboration, we have established the following agreement, and we hope you will read and comply with its terms.+Definitions++In this agreement, [FunASR Software] refers to FunASR open-source model weights and their derivatives, including finetuned models; [You] refers to individuals or organizations using, modifying, sharing, and learning from [FunASR Software].++2 License and Restrictions++2.1 License++You are free to use, copy, modify, and share [FunASR Software] under the terms of this agreement.++2.2 Restrictions++When using, copying, modifying, and sharing [FunASR Software], you must attribute the source and author information and retain relevant model names in [FunASR Software].++3 Responsibility and Risk++[FunASR Software] is provided for reference and learning purposes only, and Alibaba Group assumes no responsibility for any direct or indirect losses resulting from your use or modification of [FunASR Software]. You should assume all risks associated with using and modifying [FunASR Software].++4 Community Conduct Guidelines++4.1 Encouraged Behavior++The community welcomes developers and users to engage in discussions about [FunASR Software]. Participants are encouraged to interact in a friendly, polite, and respectful manner to foster constructive discussion and collaboration.++4.2 Prohibited Behavior++Individual or organizational users shall not engage in unjustified denigration, malicious smearing, or baseless insults against [FunASR Software]. Such behavior is considered a violation of the spirit of community cooperation. If a user is found to be engaging in the prohibited behavior mentioned above, it will be considered an automatic forfeiture of all licenses under this agreement.++5 Termination++If you violate any terms of this agreement, your license will automatically terminate, and you must cease using, copying, modifying, and sharing [FunASR Software].++6 Revisions++This agreement may be updated and revised occasionally. The revised agreement will be published in the official repository of [FunASR Software] and will take effect automatically. Continuing to use, copy, modify, and share [FunASR Software] indicates your acceptance of the revised agreement.++7 Miscellaneous++This agreement is governed by the laws of [Country/Region]. If any provision is deemed illegal, invalid, or unenforceable, that provision shall be considered severed from this agreement, and the remaining provisions shall continue to be valid and binding.++If you have any questions or comments regarding this agreement, please contact us.++Copyright © [2023-2028] [Alibaba Group]. All rights reserved.+++FunASR 模型开源协议++版本号:1.1++版权所有 (C) [2023-2028] [阿里巴巴集团]。保留所有权利。++感谢您选择 FunASR 开源模型。FunASR 开源模型包含一系列免费且开源的工业模型,让大家可以使用、修改、分享和学习该模型。++为了保证更好的社区合作,我们制定了以下协议,希望您仔细阅读并遵守本协议。++1 定义++本协议中,[FunASR 软件]指 FunASR 开源模型权重及其衍生品,包括 Finetune 后的模型;[您]指使用、修改、分享和学习[FunASR 软件]的个人或组织。++2 许可和限制++2.1 许可++您可以在遵守本协议的前提下,自由地使用、复制、修改和分享[FunASR 软件]。++2.2 限制++您在使用、复制、修改和分享[FunASR 软件]时,必须注明出处以及作者信息,并保留[FunASR 软件]中相关模型名称。++3 责任和风险承担++[FunASR 软件]仅作为参考和学习使用,不对您使用或修改[FunASR 软件]造成的任何直接或间接损失承担任何责任。您对[FunASR 软件]的使用和修改应该自行承担风险。++4 社区行为准则++4.1 欢迎交流++社区欢迎开发者与用户对[FunASR 软件]进行交流讨论。交流中请注意保持友好、礼貌和文明,以促进建设性的讨论和合作。++4.2 禁止行为++个人或组织用户不得对[FunASR 软件]进行无端诋毁、恶意抹黑或凭空谩骂。此类行为被视为违反社区合作精神。如被认定从事上述禁止行为,将视为自动放弃本协议下的所有许可。++5 终止++如果您违反本协议的任何条款,您的许可将自动终止,您必须停止使用、复制、修改和分享[FunASR 软件]。++6 修订++本协议可能会不时更新和修订。修订后的协议将在[FunASR 软件]官方仓库发布,并自动生效。如果您继续使用、复制、修改和分享[FunASR 软件],即表示您同意修订后的协议。++7 其他规定++本协议受到[国家/地区] 的法律管辖。如果任何条款被裁定为不合法、无效或无法执行,则该条款应被视为从本协议中删除,而其余条款应继续有效并具有约束力。++如果您对本协议有任何问题或意见,请联系我们。++版权所有© [2023-2028] [阿里巴巴集团]。保留所有权利。diff --git a/freevideo_engine/licenses/SenseVoice-NOTICE.txt b/freevideo_engine/licenses/SenseVoice-NOTICE.txtnew file mode 100644index 0000000..4999fe9--- /dev/null+++ b/freevideo_engine/licenses/SenseVoice-NOTICE.txt@@ -0,0 +1,14 @@+Multilingual speech recognition model for automatic subtitles++Original work: SenseVoiceSmall by FunASR / FunAudioLLM (Alibaba Group)+https://huggingface.co/FunAudioLLM/SenseVoiceSmall+Copyright (C) Alibaba Group+Licensed under the FunASR Model Open Source License Agreement, version 1.1+(full text: FunASR-MODEL_LICENSE-v1.1.txt, shipped next to this file)++Changes: the model was exported to ONNX and quantized to int8 by the sherpa-onnx+project (csukuangfj/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17). FreeVideo+uses that file unchanged and keeps the model name "SenseVoice".++The model is provided as is, without warranty of any kind. This is not an official+Alibaba Group release and Alibaba Group does not endorse FreeVideo.diff --git a/freevideo_engine/subtitle_pack.json b/freevideo_engine/subtitle_pack.jsonindex 83f40dc..62c23c1 100644--- a/freevideo_engine/subtitle_pack.json+++ b/freevideo_engine/subtitle_pack.json@@ -2,7 +2,7 @@"format": 1,"repo": "OpenVDN/vdn-minimax-h3-edge","prefix": "subtitles-v1",- "revision": null,+ "revision": "94f5ccc7bb78917e1eed47de4fc04a61465b8673","python": "cp312","groups": {"speech": [@@ -47,6 +47,26 @@"file": "licenses/Whisper-MIT.txt","bytes": 1063,"sha256": "b5d65a59060e68c4ff940e1eddfa6f94b2d68fdf58ed7f4dd57721c997e35e9d"+ },+ {+ "file": "sensevoice/model.int8.onnx",+ "bytes": 239233841,+ "sha256": "c71f0ce00bec95b07744e116345e33d8cbbe08cef896382cf907bf4b51a2cd51"+ },+ {+ "file": "sensevoice/tokens.txt",+ "bytes": 315894,+ "sha256": "f449eb28dc567533d7fa59be34e2abca8784f771850c78a47fb731a31429a1dc"+ },+ {+ "file": "licenses/FunASR-MODEL_LICENSE-v1.1.txt",+ "bytes": 5306,+ "sha256": "7dba975a2069691db4992b0592d70828b330d2f8a30a71450f4e152a554e84f8"+ },+ {+ "file": "licenses/SenseVoice-NOTICE.txt",+ "bytes": 739,+ "sha256": "3b699b8c5d3c40d0d2d3995048dc938cd9ef928d50b9550ba265c221ba8be20a"}],"runtime": [@@ -81,7 +101,7 @@},"origin": {"speech": "csukuangfj/sherpa-onnx-nemo-ctc-en-conformer-small (NVIDIA NeMo stt_en_conformer_ctc_small, CC-BY-4.0); snakers4/silero-vad v5.0 (MIT)",- "language": "csukuangfj/sherpa-onnx-whisper-tiny (OpenAI Whisper tiny, MIT)",+ "language": "csukuangfj/sherpa-onnx-whisper-tiny (OpenAI Whisper tiny, MIT); csukuangfj/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17 (SenseVoiceSmall by FunASR/FunAudioLLM, FunASR Model Open Source License Agreement v1.1)","runtime": "k2-fsa/sherpa-onnx 1.13.8 built by FreeVideo without text-to-speech (source 11afbd009a7f8c08f4bcf2fc1b265d0df4670fbf)"}}diff --git a/freevideo_engine/subtitle_sherpa.py b/freevideo_engine/subtitle_sherpa.pyindex 927d4c1..f4fecfa 100644--- a/freevideo_engine/subtitle_sherpa.py+++ b/freevideo_engine/subtitle_sherpa.py@@ -18,7 +18,9 @@ identification, `subtitles/lid/`, size and SHA-256 pinned). A speech languagewithout an installed model is refused with the `language` reason; without theverified identification files `auto` is refused as `no_engine`, so a sound inanother language never becomes a track of wrong words. An explicit language-needs its own model, never another one.+needs its own model, never another one. The multilingual model+(`subtitles/sensevoice/`, SenseVoice-Small) reads Chinese, Japanese, Korean and+Cantonese; English keeps its own model."""import hashlibimport logging@@ -35,8 +37,11 @@ CORE = 24.0 # seconds each window ownsOVERLAP = 2.0 # heard on both sides of a seam, owned by one sideTHRESHOLD = .72 # share of a line's words that must be heard in orderPAUSE = .7 # silence that ends a recognised line (seconds)-LINE_UNITS = {'zh': 24, 'en': 14}+LINE_UNITS = {'zh': 24, 'en': 14, 'yue': 24, 'ja': 20, 'ko': 10}MODELS = {'zh': 'zh', 'en': 'en'}+# SenseVoice-Small (one multilingual model) reads these languages; English keeps its own model.+MULTILINGUAL = 'sensevoice'+MULTILINGUAL_LANGUAGES = ('zh', 'ja', 'ko', 'yue')VAD_MODEL = 'silero_vad_v5.onnx'# Official Silero v5.0 export. The native VAD loader exits the process for some# incompatible exports (including faster-whisper's v6); verify before loading.@@ -54,8 +59,19 @@ LID_MIN_SECONDS = .5 # less speech than this is "nobody said anything"VAD_PAD = .4 # include soft onsets and the delay of CTC token peaks_LOG = logging.getLogger(__name__)-_WORD = re.compile(r"[㐀-鿿]|[^\W_㐀-鿿]+(?:['’][^\W_㐀-鿿]+)*", re.UNICODE)+_WORD = re.compile(r"[㐀-鿿-ヿ]|[^\W_㐀-鿿-ヿ]+(?:['’][^\W_㐀-鿿-ヿ]+)*", re.UNICODE)_CJK = re.compile(r'[㐀-鿿]')+_KANA = re.compile(r'[-ヿ]')+_HANGUL = re.compile(r'[가-]')+++def written_language(text):+ """The language a given line is written in, from its script alone."""+ if _KANA.search(text):+ return 'ja'+ if _HANGUL.search(text):+ return 'ko'+ return 'zh' if _CJK.search(text) else 'en'def units(text):@@ -264,7 +280,7 @@ def heard_words(tokens, timestamps, duration, offset=0.0):if raw.startswith(('▁', 'Ġ')) or raw in (' ', '|'):flush()for ch in raw.replace('▁', ' ').replace('Ġ', ' ').replace('|', ' '):- if '㐀' <= ch <= '鿿':+ if '㐀' <= ch <= '鿿' or '' <= ch <= 'ヿ':flush()result.append({'text': ch, 'start': piece['start'], 'end': piece['end']})elif ch.isalnum() or ch in "'’":@@ -355,7 +371,7 @@ class Engine:if language in self._directories:return languageif language == 'auto' and text.strip():- wanted = 'zh' if _CJK.search(text) else 'en'+ wanted = written_language(text)if wanted in self._directories:return wantedraise EarsError('no_engine')@@ -437,6 +453,14 @@ class Engine:if language not in self._recognizers:import sherpa_onnxfolder = self._directories[language]+ if folder.name == MULTILINGUAL:+ # One recogniser for all its languages; it names the language of each stretch itself.+ key = MULTILINGUAL+ if key not in self._recognizers:+ self._recognizers[key] = sherpa_onnx.OfflineRecognizer.from_sense_voice(+ model=str(folder / 'model.int8.onnx'), tokens=str(folder / 'tokens.txt'),+ num_threads=self._threads, use_itn=True, language='auto', provider='cpu')+ return self._recognizers[key]make = (sherpa_onnx.OfflineRecognizer.from_zipformer_ctc if language == 'zh'else sherpa_onnx.OfflineRecognizer.from_nemo_ctc)self._recognizers[language] = make(@@ -549,6 +573,10 @@ def folders(root=None):folder = base / nameif (folder / 'model.int8.onnx').is_file() and (folder / 'tokens.txt').is_file():found[language] = folder+ folder = base / MULTILINGUAL+ if (folder / 'model.int8.onnx').is_file() and (folder / 'tokens.txt').is_file():+ for language in MULTILINGUAL_LANGUAGES:+ found.setdefault(language, folder)return founddiff --git a/web/editor_text.js b/web/editor_text.jsindex 2da5d69..9ac5901 100644--- a/web/editor_text.js+++ b/web/editor_text.js@@ -53,7 +53,7 @@ export function createEditorText({element,state,t,actions,adjust,player,toolbar,function speechLabel(fraction){speech.shown=fraction;const p=Math.round(Math.min(1,Math.max(0,Number(fraction)||0))*100),size=Math.max(1,Math.round(speech.need/1e6)),text=fraction==null?(speech.need?t(`Auto subtitles · Download ${size} MB`,`自动字幕 · 下载 ${size} MB`):t('Auto subtitles','自动字幕')):speech.loading?t(`Cancel · Downloading ${p}%`,`取消 · 下载 ${p}%`):t(`Cancel · ${p}%`,`取消 · ${p}%`);for(const b of speech.buttons){b.textContent=text;b.hidden=!(speech.available||speech.need);}}function speechEnd(text){speech.stop?.();speech.stop=null;speech.job=null;speech.busy=false;speech.loading=false;speechLabel(null);if(text)actions.notify?.(text);}const speechFailed=()=>t('Could not create subtitles','没能生成字幕'),speechSilent=()=>t('No speech found','没有听到说话声');- const speechReason=reason=>reason==='no_sound'?speechSilent():reason==='language'?t('English only for now','暂时只能识别英语'):speechFailed();+ const speechReason=reason=>reason==='no_sound'?speechSilent():reason==='language'?t('Language not supported yet','暂不支持这种语言'):speechFailed();function speechResult(job){if(job.state==='running'){speechLabel(job.fraction);return true;}if(job.state!=='done'){speechEnd(job.state==='cancelled'?null:speechReason(job.reason));return false;}