Skip to content

Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.

Subtitles: multilingual speech model in the download list, notices

Bowen Xue482308c3d4b1
diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md
index ed8f93a..38c8576 100644
--- a/THIRD_PARTY_NOTICES.md
+++ b/THIRD_PARTY_NOTICES.md
@@ -19,6 +19,7 @@ FreeVideo code uses [Apache-2.0](LICENSE). Its dependencies and models retain th
| [Silero VAD](https://github.com/snakers4/silero-vad) v5.0 voice-activity model, © 2020-present Silero Team (optional download; license file downloaded with the model) | [MIT](freevideo_engine/licenses/Silero-VAD-MIT.txt) |
| [NVIDIA NeMo stt_en_conformer_ctc_small](https://huggingface.co/nvidia/stt_en_conformer_ctc_small), © NVIDIA, exported to ONNX and quantized to int8 by the sherpa-onnx project, changed from the original; not endorsed by NVIDIA (optional download for English subtitles; [notice](freevideo_engine/licenses/NeMo-stt_en_conformer_ctc_small-NOTICE.txt)) | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) |
| [OpenAI Whisper tiny](https://github.com/openai/whisper), © 2022 OpenAI, exported to ONNX and quantized to int8 by the sherpa-onnx project (optional download for spoken-language detection) | [MIT](freevideo_engine/licenses/Whisper-MIT.txt) |
+| [SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) by FunASR / FunAudioLLM (Alibaba Group), converted to ONNX and quantized to int8 by the sherpa-onnx project, used unchanged under its original name (optional download for multilingual subtitles; [notice](freevideo_engine/licenses/SenseVoice-NOTICE.txt)) | [FunASR Model Open Source License Agreement v1.1](freevideo_engine/licenses/FunASR-MODEL_LICENSE-v1.1.txt) |
| [Lucide](https://lucide.dev) icons (image editor: eyedropper, hand, crop) | [ISC](freevideo_engine/licenses/Lucide-ISC.txt) |
| [Noto Sans CJK SC and Noto Sans Mono CJK SC](https://github.com/notofonts/noto-cjk) 2.004, © 2014-2021 Adobe: subsets renamed FreeVideo Sans SC and FreeVideo Mono SC (image editor text, editor text and subtitles / 剪辑视图的文字和字幕, share card, Linux launcher) | [SIL OFL 1.1](freevideo_engine/licenses/Noto-CJK-Sans-OFL.txt) |
| [Noto Serif CJK SC](https://github.com/notofonts/noto-cjk) 2.003, © 2017-2024 Adobe: subset renamed FreeVideo Serif SC (image editor text, editor text and subtitles / 剪辑视图的文字和字幕) | [SIL OFL 1.1](freevideo_engine/licenses/Noto-CJK-Serif-OFL.txt) |
diff --git a/freevideo_engine/licenses/FunASR-MODEL_LICENSE-v1.1.txt b/freevideo_engine/licenses/FunASR-MODEL_LICENSE-v1.1.txt
new file mode 100644
index 0000000..d5cc52a
--- /dev/null
+++ b/freevideo_engine/licenses/FunASR-MODEL_LICENSE-v1.1.txt
@@ -0,0 +1,107 @@
+FunASR Model Open Source License Agreement
+
+Version: 1.1
+
+Copyright (C) [2023-2028] [Alibaba Group]. All rights reserved.
+
+Thank you for choosing the FunASR open-source model. The FunASR open-source model includes a range of free and open industrial models for you to use, modify, share, and learn from.
+
+To ensure better community collaboration, we have established the following agreement, and we hope you will read and comply with its terms.
+Definitions
+
+In this agreement, [FunASR Software] refers to FunASR open-source model weights and their derivatives, including finetuned models; [You] refers to individuals or organizations using, modifying, sharing, and learning from [FunASR Software].
+
+2 License and Restrictions
+
+2.1 License
+
+You are free to use, copy, modify, and share [FunASR Software] under the terms of this agreement.
+
+2.2 Restrictions
+
+When using, copying, modifying, and sharing [FunASR Software], you must attribute the source and author information and retain relevant model names in [FunASR Software].
+
+3 Responsibility and Risk
+
+[FunASR Software] is provided for reference and learning purposes only, and Alibaba Group assumes no responsibility for any direct or indirect losses resulting from your use or modification of [FunASR Software]. You should assume all risks associated with using and modifying [FunASR Software].
+
+4 Community Conduct Guidelines
+
+4.1 Encouraged Behavior
+
+The community welcomes developers and users to engage in discussions about [FunASR Software]. Participants are encouraged to interact in a friendly, polite, and respectful manner to foster constructive discussion and collaboration.
+
+4.2 Prohibited Behavior
+
+Individual or organizational users shall not engage in unjustified denigration, malicious smearing, or baseless insults against [FunASR Software]. Such behavior is considered a violation of the spirit of community cooperation. If a user is found to be engaging in the prohibited behavior mentioned above, it will be considered an automatic forfeiture of all licenses under this agreement.
+
+5 Termination
+
+If you violate any terms of this agreement, your license will automatically terminate, and you must cease using, copying, modifying, and sharing [FunASR Software].
+
+6 Revisions
+
+This agreement may be updated and revised occasionally. The revised agreement will be published in the official repository of [FunASR Software] and will take effect automatically. Continuing to use, copy, modify, and share [FunASR Software] indicates your acceptance of the revised agreement.
+
+7 Miscellaneous
+
+This agreement is governed by the laws of [Country/Region]. If any provision is deemed illegal, invalid, or unenforceable, that provision shall be considered severed from this agreement, and the remaining provisions shall continue to be valid and binding.
+
+If you have any questions or comments regarding this agreement, please contact us.
+
+Copyright © [2023-2028] [Alibaba Group]. All rights reserved.
+
+
+FunASR 模型开源协议
+
+版本号:1.1
+
+版权所有 (C) [2023-2028] [阿里巴巴集团]。保留所有权利。
+
+感谢您选择 FunASR 开源模型。FunASR 开源模型包含一系列免费且开源的工业模型,让大家可以使用、修改、分享和学习该模型。
+
+为了保证更好的社区合作,我们制定了以下协议,希望您仔细阅读并遵守本协议。
+
+1 定义
+
+本协议中,[FunASR 软件]指 FunASR 开源模型权重及其衍生品,包括 Finetune 后的模型;[您]指使用、修改、分享和学习[FunASR 软件]的个人或组织。
+
+2 许可和限制
+
+2.1 许可
+
+您可以在遵守本协议的前提下,自由地使用、复制、修改和分享[FunASR 软件]。
+
+2.2 限制
+
+您在使用、复制、修改和分享[FunASR 软件]时,必须注明出处以及作者信息,并保留[FunASR 软件]中相关模型名称。
+
+3 责任和风险承担
+
+[FunASR 软件]仅作为参考和学习使用,不对您使用或修改[FunASR 软件]造成的任何直接或间接损失承担任何责任。您对[FunASR 软件]的使用和修改应该自行承担风险。
+
+4 社区行为准则
+
+4.1 欢迎交流
+
+社区欢迎开发者与用户对[FunASR 软件]进行交流讨论。交流中请注意保持友好、礼貌和文明,以促进建设性的讨论和合作。
+
+4.2 禁止行为
+
+个人或组织用户不得对[FunASR 软件]进行无端诋毁、恶意抹黑或凭空谩骂。此类行为被视为违反社区合作精神。如被认定从事上述禁止行为,将视为自动放弃本协议下的所有许可。
+
+5 终止
+
+如果您违反本协议的任何条款,您的许可将自动终止,您必须停止使用、复制、修改和分享[FunASR 软件]。
+
+6 修订
+
+本协议可能会不时更新和修订。修订后的协议将在[FunASR 软件]官方仓库发布,并自动生效。如果您继续使用、复制、修改和分享[FunASR 软件],即表示您同意修订后的协议。
+
+7 其他规定
+
+本协议受到[国家/地区] 的法律管辖。如果任何条款被裁定为不合法、无效或无法执行,则该条款应被视为从本协议中删除,而其余条款应继续有效并具有约束力。
+
+如果您对本协议有任何问题或意见,请联系我们。
+
+版权所有© [2023-2028] [阿里巴巴集团]。保留所有权利。
diff --git a/freevideo_engine/licenses/SenseVoice-NOTICE.txt b/freevideo_engine/licenses/SenseVoice-NOTICE.txt
new file mode 100644
index 0000000..4999fe9
--- /dev/null
+++ b/freevideo_engine/licenses/SenseVoice-NOTICE.txt
@@ -0,0 +1,14 @@
+Multilingual speech recognition model for automatic subtitles
+
+Original work: SenseVoiceSmall by FunASR / FunAudioLLM (Alibaba Group)
+https://huggingface.co/FunAudioLLM/SenseVoiceSmall
+Copyright (C) Alibaba Group
+Licensed under the FunASR Model Open Source License Agreement, version 1.1
+(full text: FunASR-MODEL_LICENSE-v1.1.txt, shipped next to this file)
+
+Changes: the model was exported to ONNX and quantized to int8 by the sherpa-onnx
+project (csukuangfj/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17). FreeVideo
+uses that file unchanged and keeps the model name "SenseVoice".
+
+The model is provided as is, without warranty of any kind. This is not an official
+Alibaba Group release and Alibaba Group does not endorse FreeVideo.
diff --git a/freevideo_engine/subtitle_pack.json b/freevideo_engine/subtitle_pack.json
index 83f40dc..62c23c1 100644
--- a/freevideo_engine/subtitle_pack.json
+++ b/freevideo_engine/subtitle_pack.json
@@ -2,7 +2,7 @@
"format": 1,
"repo": "OpenVDN/vdn-minimax-h3-edge",
"prefix": "subtitles-v1",
- "revision": null,
+ "revision": "94f5ccc7bb78917e1eed47de4fc04a61465b8673",
"python": "cp312",
"groups": {
"speech": [
@@ -47,6 +47,26 @@
"file": "licenses/Whisper-MIT.txt",
"bytes": 1063,
"sha256": "b5d65a59060e68c4ff940e1eddfa6f94b2d68fdf58ed7f4dd57721c997e35e9d"
+ },
+ {
+ "file": "sensevoice/model.int8.onnx",
+ "bytes": 239233841,
+ "sha256": "c71f0ce00bec95b07744e116345e33d8cbbe08cef896382cf907bf4b51a2cd51"
+ },
+ {
+ "file": "sensevoice/tokens.txt",
+ "bytes": 315894,
+ "sha256": "f449eb28dc567533d7fa59be34e2abca8784f771850c78a47fb731a31429a1dc"
+ },
+ {
+ "file": "licenses/FunASR-MODEL_LICENSE-v1.1.txt",
+ "bytes": 5306,
+ "sha256": "7dba975a2069691db4992b0592d70828b330d2f8a30a71450f4e152a554e84f8"
+ },
+ {
+ "file": "licenses/SenseVoice-NOTICE.txt",
+ "bytes": 739,
+ "sha256": "3b699b8c5d3c40d0d2d3995048dc938cd9ef928d50b9550ba265c221ba8be20a"
}
],
"runtime": [
@@ -81,7 +101,7 @@
},
"origin": {
"speech": "csukuangfj/sherpa-onnx-nemo-ctc-en-conformer-small (NVIDIA NeMo stt_en_conformer_ctc_small, CC-BY-4.0); snakers4/silero-vad v5.0 (MIT)",
- "language": "csukuangfj/sherpa-onnx-whisper-tiny (OpenAI Whisper tiny, MIT)",
+ "language": "csukuangfj/sherpa-onnx-whisper-tiny (OpenAI Whisper tiny, MIT); csukuangfj/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17 (SenseVoiceSmall by FunASR/FunAudioLLM, FunASR Model Open Source License Agreement v1.1)",
"runtime": "k2-fsa/sherpa-onnx 1.13.8 built by FreeVideo without text-to-speech (source 11afbd009a7f8c08f4bcf2fc1b265d0df4670fbf)"
}
}
diff --git a/freevideo_engine/subtitle_sherpa.py b/freevideo_engine/subtitle_sherpa.py
index 927d4c1..f4fecfa 100644
--- a/freevideo_engine/subtitle_sherpa.py
+++ b/freevideo_engine/subtitle_sherpa.py
@@ -18,7 +18,9 @@ identification, `subtitles/lid/`, size and SHA-256 pinned). A speech language
without an installed model is refused with the `language` reason; without the
verified identification files `auto` is refused as `no_engine`, so a sound in
another language never becomes a track of wrong words. An explicit language
-needs its own model, never another one.
+needs its own model, never another one. The multilingual model
+(`subtitles/sensevoice/`, SenseVoice-Small) reads Chinese, Japanese, Korean and
+Cantonese; English keeps its own model.
"""
import hashlib
import logging
@@ -35,8 +37,11 @@ CORE = 24.0 # seconds each window owns
OVERLAP = 2.0 # heard on both sides of a seam, owned by one side
THRESHOLD = .72 # share of a line's words that must be heard in order
PAUSE = .7 # silence that ends a recognised line (seconds)
-LINE_UNITS = {'zh': 24, 'en': 14}
+LINE_UNITS = {'zh': 24, 'en': 14, 'yue': 24, 'ja': 20, 'ko': 10}
MODELS = {'zh': 'zh', 'en': 'en'}
+# SenseVoice-Small (one multilingual model) reads these languages; English keeps its own model.
+MULTILINGUAL = 'sensevoice'
+MULTILINGUAL_LANGUAGES = ('zh', 'ja', 'ko', 'yue')
VAD_MODEL = 'silero_vad_v5.onnx'
# Official Silero v5.0 export. The native VAD loader exits the process for some
# incompatible exports (including faster-whisper's v6); verify before loading.
@@ -54,8 +59,19 @@ LID_MIN_SECONDS = .5 # less speech than this is "nobody said anything"
VAD_PAD = .4 # include soft onsets and the delay of CTC token peaks
_LOG = logging.getLogger(__name__)
-_WORD = re.compile(r"[㐀-鿿]|[^\W_㐀-鿿]+(?:['’][^\W_㐀-鿿]+)*", re.UNICODE)
+_WORD = re.compile(r"[㐀-鿿぀-ヿ]|[^\W_㐀-鿿぀-ヿ]+(?:['’][^\W_㐀-鿿぀-ヿ]+)*", re.UNICODE)
_CJK = re.compile(r'[㐀-鿿]')
+_KANA = re.compile(r'[぀-ヿ]')
+_HANGUL = re.compile(r'[가-힯]')
+
+
+def written_language(text):
+ """The language a given line is written in, from its script alone."""
+ if _KANA.search(text):
+ return 'ja'
+ if _HANGUL.search(text):
+ return 'ko'
+ return 'zh' if _CJK.search(text) else 'en'
def units(text):
@@ -264,7 +280,7 @@ def heard_words(tokens, timestamps, duration, offset=0.0):
if raw.startswith(('▁', 'Ġ')) or raw in (' ', '|'):
flush()
for ch in raw.replace('▁', ' ').replace('Ġ', ' ').replace('|', ' '):
- if '㐀' <= ch <= '鿿':
+ if '㐀' <= ch <= '鿿' or '぀' <= ch <= 'ヿ':
flush()
result.append({'text': ch, 'start': piece['start'], 'end': piece['end']})
elif ch.isalnum() or ch in "'’":
@@ -355,7 +371,7 @@ class Engine:
if language in self._directories:
return language
if language == 'auto' and text.strip():
- wanted = 'zh' if _CJK.search(text) else 'en'
+ wanted = written_language(text)
if wanted in self._directories:
return wanted
raise EarsError('no_engine')
@@ -437,6 +453,14 @@ class Engine:
if language not in self._recognizers:
import sherpa_onnx
folder = self._directories[language]
+ if folder.name == MULTILINGUAL:
+ # One recogniser for all its languages; it names the language of each stretch itself.
+ key = MULTILINGUAL
+ if key not in self._recognizers:
+ self._recognizers[key] = sherpa_onnx.OfflineRecognizer.from_sense_voice(
+ model=str(folder / 'model.int8.onnx'), tokens=str(folder / 'tokens.txt'),
+ num_threads=self._threads, use_itn=True, language='auto', provider='cpu')
+ return self._recognizers[key]
make = (sherpa_onnx.OfflineRecognizer.from_zipformer_ctc if language == 'zh'
else sherpa_onnx.OfflineRecognizer.from_nemo_ctc)
self._recognizers[language] = make(
@@ -549,6 +573,10 @@ def folders(root=None):
folder = base / name
if (folder / 'model.int8.onnx').is_file() and (folder / 'tokens.txt').is_file():
found[language] = folder
+ folder = base / MULTILINGUAL
+ if (folder / 'model.int8.onnx').is_file() and (folder / 'tokens.txt').is_file():
+ for language in MULTILINGUAL_LANGUAGES:
+ found.setdefault(language, folder)
return found
diff --git a/web/editor_text.js b/web/editor_text.js
index 2da5d69..9ac5901 100644
--- a/web/editor_text.js
+++ b/web/editor_text.js
@@ -53,7 +53,7 @@ export function createEditorText({element,state,t,actions,adjust,player,toolbar,
function speechLabel(fraction){speech.shown=fraction;const p=Math.round(Math.min(1,Math.max(0,Number(fraction)||0))*100),size=Math.max(1,Math.round(speech.need/1e6)),text=fraction==null?(speech.need?t(`Auto subtitles · Download ${size} MB`,`自动字幕 · 下载 ${size} MB`):t('Auto subtitles','自动字幕')):speech.loading?t(`Cancel · Downloading ${p}%`,`取消 · 下载 ${p}%`):t(`Cancel · ${p}%`,`取消 · ${p}%`);for(const b of speech.buttons){b.textContent=text;b.hidden=!(speech.available||speech.need);}}
function speechEnd(text){speech.stop?.();speech.stop=null;speech.job=null;speech.busy=false;speech.loading=false;speechLabel(null);if(text)actions.notify?.(text);}
const speechFailed=()=>t('Could not create subtitles','没能生成字幕'),speechSilent=()=>t('No speech found','没有听到说话声');
- const speechReason=reason=>reason==='no_sound'?speechSilent():reason==='language'?t('English only for now','暂时只能识别英语'):speechFailed();
+ const speechReason=reason=>reason==='no_sound'?speechSilent():reason==='language'?t('Language not supported yet','暂不支持这种语言'):speechFailed();
function speechResult(job){
if(job.state==='running'){speechLabel(job.fraction);return true;}
if(job.state!=='done'){speechEnd(job.state==='cancelled'?null:speechReason(job.reason));return false;}