Skip to content

Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.

Editor: music for a stretch of the second sound track from the generation bar, lowered during speech

Bowen Xuecf720c649d81
diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md
index 66b947e..5f7e5ba 100644
--- a/THIRD_PARTY_NOTICES.md
+++ b/THIRD_PARTY_NOTICES.md
@@ -19,7 +19,7 @@ FreeVideo code uses [Apache-2.0](LICENSE). Its dependencies and models retain th
| [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) 1.13.8 speech runtime, built by FreeVideo without text-to-speech, ALSA and PortAudio (modified build; `__init__.py` edited; source snapshot 11afbd00 kept by FreeVideo; optional download for automatic subtitles, [build note](freevideo_engine/licenses/sherpa-onnx-build-NOTE.txt)). Includes statically linked kaldi-native-fbank, kaldi-decoder, kaldifst, OpenFst, simple-sentencepiece (Apache-2.0), KissFFT (BSD-3-Clause), pybind11 (BSD-3-Clause), nlohmann/json (MIT), hclust-cpp and Darts-clone (BSD-2-Clause), ThreadPool (Zlib). The Windows build also contains msvcp140.dll and msvcp140_1.dll (Microsoft Visual C++ 2022 runtime 14.44.35211, © Microsoft Corporation), unmodified and redistributed as permitted by Microsoft for its redistributable files. | Apache-2.0 |
| [ONNX Runtime](https://github.com/microsoft/onnxruntime) 1.28.2 (CPU), © Microsoft Corporation; its third-party notices are shipped inside the sherpa-onnx wheel | MIT |
| [Eigen](https://gitlab.com/libeigen/eigen) 5.0.1, compiled into sherpa-onnx and ONNX Runtime; source: https://gitlab.com/libeigen/eigen/-/tree/5.0.1 | MPL-2.0 |
-| [Silero VAD](https://github.com/snakers4/silero-vad) v5.0 voice-activity model, © 2020-present Silero Team (optional download; license file downloaded with the model) | [MIT](freevideo_engine/licenses/Silero-VAD-MIT.txt) |
+| [Silero VAD](https://github.com/snakers4/silero-vad) v5.0 voice-activity model, © 2020-present Silero Team (optional download with subtitles, license file downloaded with the model; the onnx-community export is downloaded with the music model) | [MIT](freevideo_engine/licenses/Silero-VAD-MIT.txt) |
| [NVIDIA NeMo stt_en_conformer_ctc_small](https://huggingface.co/nvidia/stt_en_conformer_ctc_small), © NVIDIA, exported to ONNX and quantized to int8 by the sherpa-onnx project, changed from the original; not endorsed by NVIDIA (optional download for English subtitles; [notice](freevideo_engine/licenses/NeMo-stt_en_conformer_ctc_small-NOTICE.txt)) | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) |
| [OpenAI Whisper tiny](https://github.com/openai/whisper), © 2022 OpenAI, exported to ONNX and quantized to int8 by the sherpa-onnx project (optional download for spoken-language detection) | [MIT](freevideo_engine/licenses/Whisper-MIT.txt) |
| [SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) by FunASR / FunAudioLLM (Alibaba Group), converted to ONNX and quantized to int8 by the sherpa-onnx project, used unchanged under its original name (optional download for multilingual subtitles; [notice](freevideo_engine/licenses/SenseVoice-NOTICE.txt)) | [FunASR Model Open Source License Agreement v1.1](freevideo_engine/licenses/FunASR-MODEL_LICENSE-v1.1.txt) |
diff --git a/__init__.py b/__init__.py
index 886ebff..3349199 100644
--- a/__init__.py
+++ b/__init__.py
@@ -45,6 +45,8 @@ async def comfy_entrypoint():
capabilities_routes()
from .freevideo_engine.subtitle_ears import register as subtitle_ears_routes
subtitle_ears_routes()
+ from .freevideo_engine.editor_music import register as editor_music_routes
+ editor_music_routes()
from .freevideo_engine.subtitle_files import register as subtitle_file_routes
subtitle_file_routes()
from .freevideo_engine.comfy_share import register as share_routes
diff --git a/freevideo_engine/editor_audio.py b/freevideo_engine/editor_audio.py
index d65a2af..ab605d8 100644
--- a/freevideo_engine/editor_audio.py
+++ b/freevideo_engine/editor_audio.py
@@ -18,6 +18,17 @@ def clip_fields(raw, kind):
or value == raw['id'] or raw.get('link')):
raise ValueError('Invalid detached audio')
result['detached_from'] = value
+ # A music piece's preset (prefilled when it is made again) and whether it is lowered under the first sound track.
+ if 'music' in raw:
+ value = raw['music']
+ if (kind != 'audio' or not isinstance(value, dict) or set(value) != {'style', 'mood'}
+ or not all(isinstance(value[k], str) and re.fullmatch(r'[a-z]{1,20}', value[k]) for k in ('style', 'mood'))):
+ raise ValueError('Invalid music preset')
+ result['music'] = {'style': value['style'], 'mood': value['mood']}
+ if 'duck' in raw:
+ if kind != 'audio' or type(raw['duck']) is not bool:
+ raise ValueError('Invalid audio ducking')
+ result['duck'] = raw['duck']
return result
@@ -157,3 +168,82 @@ def apply_track_audio(block, block_start, track, tracks, *, rate=48000, fps=24,
added[:, a - block_start:b - block_start] += (data[:, offset + a - begin:offset + b - begin]
* gain(np.arange(a, b))[None, :] * clip.get('volume', 1))
return output + added
+
+
+# Lower during speech: a clip with duck=True sits DUCK_DB lower wherever another audible clip has speech,
+# with DUCK_RAMP_SECONDS ramps just outside each stretch. web/editor_audio_model.js has the same rules.
+DUCK_DB = -8.
+DUCK_RAMP_SECONDS = .3
+
+
+def _speed(clip):
+ return clip.get('speed', 1) if changed(clip) else 1
+
+
+def duck_ranges(tracks, spans, rate=48000, fps=24):
+ """Timeline sample ranges [first, last) with speech in a clip that is not itself lowered.
+
+ spans maps 'kind:id' source keys to [start, end] seconds of that source; a source without an entry
+ has no known speech. Ranges closer than two ramps are joined so the level does not bounce between them.
+ """
+ found = []
+ for track in tracks:
+ if track.get('kind') != 'audio' or not track_audible(track, tracks):
+ continue
+ for clip in track['clips']:
+ source = clip.get('source')
+ if not source or clip.get('duck') or clip.get('volume', 1) == 0 or not clip_audible(clip, track, tracks):
+ continue
+ stretches = spans.get(source['kind'] + ':' + source['id']) or ()
+ first, last = _samples(clip['start'], rate, fps), _samples(clip['start'] + clip['frames'], rate, fps)
+ origin = (source_sample_start(clip) * rate / 48000 if audio_changed(clip)
+ else _source_start(clip, rate, fps))
+ speed = _speed(clip)
+ for begin, end in stretches:
+ a, b = (begin * rate - origin) / speed, (end * rate - origin) / speed
+ if clip.get('reverse'):
+ a, b = (last - first) - b, (last - first) - a
+ a, b = max(first, math.floor(first + a + .5)), min(last, math.floor(first + b + .5))
+ if b > a:
+ found.append([a, b])
+ found.sort()
+ ramp = 2 * max(1, math.floor(DUCK_RAMP_SECONDS * rate + .5))
+ joined = []
+ for a, b in found:
+ if joined and a <= joined[-1][1] + ramp:
+ joined[-1][1] = max(joined[-1][1], b)
+ else:
+ joined.append([a, b])
+ return joined
+
+
+def duck_gain(ranges, block_start, count, rate=48000):
+ """Gain per sample for a lowered clip over [block_start, block_start + count)."""
+ import numpy as np
+ gain = np.ones(count, dtype=np.float64)
+ ramp = max(1, math.floor(DUCK_RAMP_SECONDS * rate + .5))
+ low = 10. ** (DUCK_DB / 20.)
+ position = np.arange(block_start, block_start + count, dtype=np.float64)
+ for first, last in ranges:
+ if last + ramp <= block_start or first - ramp >= block_start + count:
+ continue
+ into = np.clip((position - (first - ramp)) / ramp, 0, 1)
+ out_of = np.clip(((last + ramp) - position) / ramp, 0, 1)
+ np.minimum(gain, 1 - (1 - low) * np.minimum(into, out_of), out=gain)
+ return gain
+
+
+def duck_sources(tracks):
+ """Source keys whose speech spans a lowered clip needs (none when nothing is lowered)."""
+ if not any(clip.get('duck') and clip.get('source') for track in tracks for clip in track['clips']):
+ return []
+ keys = []
+ for track in tracks:
+ if track.get('kind') != 'audio':
+ continue
+ for clip in track['clips']:
+ source = clip.get('source')
+ key = source and source['kind'] + ':' + source['id']
+ if key and not clip.get('duck') and key not in keys:
+ keys.append(key)
+ return keys
diff --git a/freevideo_engine/editor_export.py b/freevideo_engine/editor_export.py
index 671ddf2..01a0659 100644
--- a/freevideo_engine/editor_export.py
+++ b/freevideo_engine/editor_export.py
@@ -28,7 +28,8 @@ from .editor_limits import PROJECT_BYTES
from .editor_text import is_text_track, has_text_data, validate_text_project, paint_project, check_export_text, MissingGlyphs
from . import editor_export_options
from .editor_speed import validate_clip, changed as speed_changed, audio_changed, independent_join, VideoReader, AudioReader
-from .editor_audio import clip_fields, track_fields, clip_audible, track_audible, crossfade_reads, apply_track_audio
+from .editor_audio import (clip_fields, track_fields, clip_audible, track_audible, crossfade_reads, apply_track_audio,
+ duck_ranges, duck_gain, duck_sources)
FPS = 24
SAMPLE_RATE = 48000
@@ -693,6 +694,10 @@ def _render_once(project, resolve, target, *, progress, cancelled, options, repa
audio_tracks = {track['id']: track for track in project['tracks'] if track['kind'] == 'audio'}
envelopes = {key: _CrossfadeReads(track, resolved.__getitem__, cancelled)
for key, track in audio_tracks.items()}
+ # Lower during speech: the same cached spans the preview reads.
+ keys = duck_sources(project['tracks'])
+ ducking = duck_ranges(project['tracks'], _speech_spans(
+ keys, lambda key: (resolved[key] if key in resolved else resolve(key)).get('audio'), cancelled)) if keys else []
def joinable(track):
# A continuation joins its sound only where nothing else shapes that seam:
@@ -800,6 +805,8 @@ def _render_once(project, resolve, target, *, progress, cancelled, options, repa
or not clip_audible(following, audio_tracks[track], project['tracks'])
or following['start'] != clip['start'] + clip['frames'])):
gain *= np.clip((clip['frames'] * SAMPLES_PER_FRAME - 1 - positions) / (FADE_SAMPLES - 1), 0, 1).astype(np.float32)
+ if clip.get('duck') and ducking:
+ gain *= duck_gain(ducking, number * SAMPLES_PER_FRAME, SAMPLES_PER_FRAME).astype(np.float32)
track_block = samples * gain[None, :] * clip.get('volume', 1)
extras = (envelopes[track].extras(number * SAMPLES_PER_FRAME, SAMPLES_PER_FRAME)
if track_audible(audio_tracks[track], project['tracks']) else None)
@@ -853,6 +860,15 @@ def _render_once(project, resolve, target, *, progress, cancelled, options, repa
target.unlink(missing_ok=True)
+def _speech_spans(keys, audio, cancelled):
+ from .editor_music import SpeechSpans
+ from .speech import SpeechError
+ try:
+ return SpeechSpans(_output_root(), resolve=audio).spans(keys, cancelled)
+ except SpeechError:
+ raise ExportCancelled() from None
+
+
def _output_root():
import folder_paths
return Path(folder_paths.get_output_directory()).resolve()
diff --git a/freevideo_engine/editor_music.py b/freevideo_engine/editor_music.py
new file mode 100644
index 0000000..d087024
--- /dev/null
+++ b/freevideo_engine/editor_music.py
@@ -0,0 +1,600 @@
+"""Music for the editing page: a stretch of the sound track made into one piece of music.
+
+The page plans the piece where the user dragged (start and length on the sound track), sends the
+length and what the music should be (its own words, or a style and a mood), and lays the finished
+file into that stretch. This module only makes the file: first-use download of the music model,
+one job at a time, progress, cancel. The music service yields to video: a video that starts while
+the music is made stops it, and the job waits for the video and composes again.
+"""
+import logging
+import re
+import secrets
+import threading
+import time
+from collections import deque
+from pathlib import Path
+
+FPS = 24
+FOLDER = 'freevideo-music'
+MAX_JOBS = 20
+MAX_QUIET = 500
+MAX_TEXT = 512
+# A video that starts while the music is made stops it; the job waits and composes again (as the Agent's job does).
+PREEMPTIONS = 8
+POLL_SECONDS = .5
+_JOB_ID = re.compile(r'm-[0-9a-f]{32}')
+_FIELDS = {'frames', 'text', 'style', 'mood', 'seed', 'quiet'}
+REASONS = ('music_download_failed', 'out_of_memory', 'music_failed', 'invalid_request', 'busy', 'unsupported', 'failed')
+
+# Presets: the genre and instruments, then the feeling and pace. Instrumental only, as the Agent asks for.
+STYLES = {
+ 'cinematic': 'Cinematic orchestral score with strings, soft brass and piano',
+ 'pop': 'Modern instrumental pop with clean electric guitar, bass, light drums and synth pads',
+ 'electronic': 'Electronic track with warm synthesizers, a steady drum machine and a deep bass line',
+ 'piano': 'Solo piano piece with a clear melody and gentle sustain',
+ 'folk': 'Acoustic folk tune with fingerpicked guitar, light percussion and a soft violin',
+ 'ambient': 'Ambient soundscape with slow evolving pads, soft textures and distant bells',
+}
+MOODS = {
+ 'calm': 'calm and peaceful, slow tempo around 70 BPM',
+ 'upbeat': 'bright and upbeat, lively tempo around 120 BPM',
+ 'tense': 'tense and suspenseful, a low pulse around 90 BPM that slowly builds',
+ 'warm': 'warm and tender, moderate tempo around 85 BPM',
+ 'sad': 'sad and reflective, slow tempo around 65 BPM in a minor key',
+}
+# The feeling alone, without the pace: used after the user's own words, which set the pace themselves
+# (a fixed "around 70 BPM" after "a festival parade" pulled the piece back to the preset).
+FEELINGS = {
+ 'calm': 'calm and peaceful',
+ 'upbeat': 'bright and upbeat',
+ 'tense': 'tense and suspenseful',
+ 'warm': 'warm and tender',
+ 'sad': 'sad and reflective, in a minor key',
+}
+DEFAULT_STYLE, DEFAULT_MOOD = 'cinematic', 'calm'
+
+
+def caption(style, mood, words=None):
+ """The English description the music model reads for a preset pair, or for the user's words with the pair after them."""
+ if style not in STYLES or mood not in MOODS:
+ raise ValueError('Unknown style or mood')
+ if words:
+ return '%s. %s, %s. Instrumental only, no singing, no vocals.' % (words.rstrip('.'), STYLES[style], FEELINGS[mood])
+ return '%s, %s. Instrumental only, no singing, no vocals.' % (STYLES[style], MOODS[mood])
+
+
+def _whole(value, low, high):
+ return type(value) is int and low <= value <= high
+
+
+def max_frames():
+ from .music import service
+ return service.MAX_SECONDS * FPS
+
+
+def validate_request(body):
+ """The page's request -> what the job runs. Unknown fields, unknown presets and bad ranges are refused."""
+ if not isinstance(body, dict) or set(body) - _FIELDS:
+ raise ValueError('Invalid request')
+ frames = body.get('frames')
+ if not _whole(frames, 1, max_frames()):
+ raise ValueError('Invalid length')
+ text = body.get('text', '')
+ if not isinstance(text, str) or len(text) > MAX_TEXT or '\x00' in text:
+ raise ValueError('Invalid description')
+ text = ' '.join(text.split())
+ style, mood = body.get('style'), body.get('mood')
+ if (style is None) != (mood is None):
+ raise ValueError('Style and mood go together')
+ if style is not None and (style not in STYLES or mood not in MOODS):
+ raise ValueError('Unknown style or mood')
+ if not text and style is None:
+ style, mood = DEFAULT_STYLE, DEFAULT_MOOD
+ seed = body.get('seed')
+ if seed is None:
+ seed = secrets.randbelow(2**32)
+ elif not _whole(seed, 0, 2**32 - 1):
+ raise ValueError('Invalid seed')
+ quiet = body.get('quiet', [])
+ if not isinstance(quiet, list) or len(quiet) > MAX_QUIET:
+ raise ValueError('Invalid quiet spans')
+ spans = []
+ for span in quiet:
+ if (not isinstance(span, dict) or set(span) != {'start', 'frames'} or not _whole(span['start'], 0, frames - 1)
+ or not _whole(span['frames'], 1, frames - span['start'])):
+ raise ValueError('Invalid quiet span')
+ spans.append((span['start'], span['frames']))
+ # The user's own words go first, with the chosen style and feeling after them (not the preset's pace);
+ # the full preset alone when there are none.
+ if text:
+ words, stem = text if style is None else caption(style, mood, text), 'text'
+ else:
+ words, stem = caption(style, mood), '%s-%s' % (style, mood)
+ return dict(frames=frames, caption=words, stem=stem, style=style, mood=mood, seed=seed, quiet=spans)
+
+
+def _gpu_busy(server):
+ """True while a video (queued or running), a picture or the prompt model has the GPU."""
+ try:
+ from . import resident_process
+ if resident_process.OWNER.active:
+ return True
+ except Exception: # noqa: BLE001 - an optional check; the music service refuses again if the GPU is taken
+ pass
+ if server is None:
+ return False
+ try:
+ front = getattr(server, '_freevideo_front', None)
+ running, pending = front.current_queue() if front is not None else server.prompt_queue.get_current_queue()
+ if running or pending:
+ return True
+ except Exception: # noqa: BLE001
+ pass
+ for name in ('_freevideo_prompt_vlm',):
+ service = getattr(server, name, None)
+ try:
+ if service is not None and service.busy():
+ return True
+ except Exception: # noqa: BLE001
+ pass
+ try:
+ from .image_gen.service import get_service
+ service = get_service(server)
+ return service is not None and service.busy()
+ except Exception: # noqa: BLE001
+ return False
+
+
+def run_job(request, *, root, out, phase, cancelled, server=None, service=None, busy=None, poll=POLL_SECONDS):
+ """Compose one piece into `out`; returns the music service's numbers. Raises InterruptedError when cancelled."""
+ if service is None:
+ from .music import service
+ busy = busy or (lambda: _gpu_busy(server))
+ for attempt in range(PREEMPTIONS + 1):
+ try:
+ return service.compose(out, request['caption'], request['frames'], seed=request['seed'], quiet=request['quiet'],
+ fps=FPS, server=server, cancelled=cancelled, phase=phase)
+ except service.Preempted:
+ if attempt == PREEMPTIONS:
+ raise service.MusicError('music_failed', 'preempted') from None
+ phase('waiting')
+ # Wait out the video (and anything queued with it) before the GPU is taken again.
+ while True:
+ if cancelled():
+ raise InterruptedError('cancelled') from None
+ time.sleep(poll)
+ if not busy():
+ break
+
+
+class MusicQueue:
+ """One worker, first in first out, cancellable; keeps the last MAX_JOBS jobs for the page to read back."""
+ def __init__(self, root, *, runner=None, read=None, clock=time.monotonic, **kwargs):
+ self.root = Path(root).absolute()
+ self.runner = runner or run_job
+ self.read = read or self._read
+ self.clock = clock
+ self.kwargs = kwargs
+ self.lock = threading.RLock()
+ self.jobs = {}
+ self.pending = deque()
+ self.worker = None
+ # Seconds of composing per second of music on this machine, from the last finished job.
+ self.pace = None
+
+ def _read(self, source):
+ from . import comfy_editor
+ return comfy_editor.sources(self.root, [source])[source]
+
+ def _view(self, job):
+ value = {key: job[key] for key in ('id', 'state', 'phase', 'fraction', 'reason', 'result')}
+ # Composing reports no steps; the page shows time instead, from how long it took here last time.
+ if job['state'] == 'running' and job['phase'] == 'composing' and job['began'] is not None:
+ value['elapsed'] = round(self.clock() - job['began'], 1)
+ value['estimate'] = None if self.pace is None else round(self.pace * job['request']['frames'] / FPS, 1)
+ return value
+
+ def submit(self, body):
+ request = validate_request(body)
+ hexid = secrets.token_hex(16)
+ job = dict(id='m-' + hexid, state='queued', phase=None, fraction=0., reason=None, result=None, began=None,
+ request=request, relative='%s/%s-%s.mp4' % (FOLDER, request['stem'], hexid), stop=threading.Event())
+ with self.lock:
+ for old in list(self.jobs):
+ if len(self.jobs) < MAX_JOBS:
+ break
+ if self.jobs[old]['state'] not in ('queued', 'running'):
+ del self.jobs[old]
+ if len(self.jobs) >= MAX_JOBS:
+ raise MusicBusy()
+ self.jobs[job['id']] = job
+ self.pending.append(job['id'])
+ initial = self._view(job)
+ if self.worker is None:
+ self.worker = threading.Thread(target=self._run, name='freevideo-editor-music', daemon=True)
+ self.worker.start()
+ return initial
+
+ def get(self, identity):
+ if not isinstance(identity, str) or not _JOB_ID.fullmatch(identity):
+ raise ValueError('Invalid job')
+ with self.lock:
+ if identity not in self.jobs:
+ raise KeyError(identity)
+ return self._view(self.jobs[identity])
+
+ def cancel(self, identity):
+ with self.lock:
+ self.get(identity)
+ job = self.jobs[identity]
+ if job['state'] in ('queued', 'running'):
+ job['stop'].set()
+ if identity in self.pending:
+ self.pending.remove(identity)
+ job.update(state='cancelled', phase=None)
+ return self._view(job)
+
+ def busy(self):
+ with self.lock:
+ return any(job['state'] in ('queued', 'running') for job in self.jobs.values())
+
+ def _run(self):
+ while True:
+ with self.lock:
+ if not self.pending:
+ self.worker = None
+ return
+ job = self.jobs[self.pending.popleft()]
+ if job['stop'].is_set():
+ job['state'] = 'cancelled'
+ continue
+ job['state'] = 'running'
+
+ def phase(name, fraction=None):
+ with self.lock:
+ if name == 'composing' and job['phase'] != 'composing':
+ job['began'] = self.clock()
+ job['phase'] = name
+ job['fraction'] = 0. if fraction is None else max(0., min(.99, float(fraction)))
+
+ out = self.root / job['relative']
+ try:
+ out.parent.mkdir(parents=True, exist_ok=True)
+ began = self.clock()
+ numbers = self.runner(job['request'], root=self.root, out=out, phase=phase, cancelled=job['stop'].is_set,
+ **self.kwargs) or {}
+ if job['stop'].is_set():
+ raise InterruptedError('cancelled')
+ source = 'generated:' + job['relative']
+ found = self.read(source)
+ if found.get('missing') or not found.get('frames') or not found.get('has_audio'):
+ raise MusicFailed('music_failed')
+ spent = numbers.get('load_s', 0) + numbers.get('generate_s', 0) if numbers.get('generate_s') else self.clock() - began
+ with self.lock:
+ self.pace = max(.05, spent / (job['request']['frames'] / FPS))
+ job.update(state='done', phase=None, fraction=1., result=dict(
+ source=source, frames=int(found['frames']), seed=job['request']['seed'],
+ style=job['request']['style'], mood=job['request']['mood']))
+ except BaseException as error: # noqa: BLE001 - every outcome is the job's; the worker keeps serving
+ out.unlink(missing_ok=True)
+ with self.lock:
+ if isinstance(error, InterruptedError) or job['stop'].is_set():
+ job.update(state='cancelled', phase=None, reason=None)
+ else:
+ code = getattr(error, 'code', None)
+ if code not in REASONS:
+ logging.warning('[FreeVideo] Editor music failed', exc_info=error)
+ code = 'music_failed'
+ job.update(state='failed', phase=None, reason=code)
+ if not isinstance(error, Exception):
+ raise
+
+
+class MusicBusy(Exception):
+ code = 'busy'
+
+
+class MusicFailed(Exception):
+ def __init__(self, code):
+ super().__init__(code)
+ self.code = code
+
+
+class MusicPack:
+ """First-use download of the music model, with the download sources chosen in Settings."""
+ def __init__(self, *, installation=None, service=None, status=None, free_bytes=None):
+ self._installation = installation
+ self._service = service
+ self._status = status
+ self._free = free_bytes
+ self.lock = threading.Lock()
+ self.value = dict(running=False, fraction=0., reason=None)
+ self.stop = threading.Event()
+ self.worker = None
+
+ def installation(self):
+ if self._installation is not None:
+ return self._installation()
+ from .comfy_bridge import installation
+ return installation()
+
+ def service(self):
+ if self._service is not None:
+ return self._service
+ from .music import service
+ return service
+
+ def status(self):
+ root, machine = self.installation()
+ if self._status is not None:
+ pack = self._status(machine['model_root'])
+ else:
+ from .image_gen import image_pack
+ pack = image_pack.status(machine['model_root'], group='music')
+ if self._free is not None:
+ free = self._free(machine['model_root'])
+ else:
+ from . import disk_space
+ from .image_gen import image_pack
+ folder = image_pack.base(machine['model_root'])
+ while not folder.exists() and folder.parent != folder:
+ folder = folder.parent
+ free = disk_space.free_bytes(folder)
+ remaining = int(pack.get('remaining_bytes', pack.get('bytes', 0)))
+ from .image_gen.image_pack import RESERVE
+ with self.lock:
+ value = dict(self.value)
+ value.update(ready=bool(pack['ready']), bytes=remaining,
+ disk_ok=bool(pack['ready']) or free >= remaining + RESERVE)
+ return value
+
+ def start(self):
+ """Begin the download unless it runs already; returns False when the disk cannot take it."""
+ value = self.status()
+ if value['ready'] or value['running']:
+ return True
+ if not value['disk_ok']:
+ with self.lock:
+ self.value.update(reason='disk_space')
+ return False
+ root, machine = self.installation()
+ with self.lock:
+ if self.value['running']:
+ return True
+ self.stop = threading.Event()
+ self.value = dict(running=True, fraction=0., reason=None)
+ self.worker = threading.Thread(target=self._run, args=(root, machine['model_root'], self.stop),
+ name='freevideo-editor-music-pack', daemon=True)
+ self.worker.start()
+ return True
+
+ def cancel(self):
+ self.stop.set()
+
+ def _run(self, root, model_root, stop):
+ service = self.service()
+
+ def progress(fraction):
+ with self.lock:
+ self.value['fraction'] = max(self.value['fraction'], min(.99, float(fraction)))
+
+ reason = None
+ try:
+ service.install(model_root, root, progress, stop.is_set)
+ except InterruptedError:
+ pass
+ except service.MusicError as error:
+ reason = error.reason or 'failed'
+ except Exception: # noqa: BLE001 - the page says it did not download; the log keeps why
+ logging.warning('[FreeVideo] Music model download failed', exc_info=True)
+ reason = 'failed'
+ with self.lock:
+ self.value = dict(running=False, fraction=0. if reason or stop.is_set() else 1., reason=reason)
+
+
+def available(installation=None):
+ """The music model runs where video runs through CUDA (NVIDIA and AMD ROCm); not on Apple silicon yet."""
+ try:
+ if installation is None:
+ from .comfy_bridge import installation
+ _root, machine = installation()
+ return machine.get('device_backend', 'cuda') != 'mps'
+ except Exception: # noqa: BLE001 - no engine, no music
+ return False
+
+
+class SpeechSpans:
+ """Speech spans per editor source for 'Lower during speech'; one file at a time, off the request thread.
+
+ The preview asks per source and gets pending until the spans are ready; the export asks for all of its
+ sources at once and waits. Both read the same cached spans for the same file.
+ """
+ def __init__(self, root, *, resolve=None, model_path=None, compute=None):
+ self.root = Path(root)
+ self._resolve, self._model_path, self._compute = resolve, model_path, compute
+ self.lock = threading.Lock()
+ self.pending, self.failed, self.known = deque(), {}, {}
+ self.worker = None
+
+ def cache(self):
+ return self.root / 'FreeVideo' / '.editor' / 'speech'
+
+ def model_path(self):
+ if self._model_path is not None:
+ return Path(self._model_path)
+ from .comfy_bridge import installation
+ from .image_gen import image_pack
+ from .speech import MODEL_FILE
+ return image_pack.base(installation()[1]['model_root']) / MODEL_FILE
+
+ def audio(self, key):
+ if self._resolve is not None:
+ return self._resolve(key)
+ from .editor_export import resolve_source
+ return resolve_source(key).get('audio')
+
+ def compute(self, path, cancelled=None):
+ if self._compute is not None:
+ return self._compute(path)
+ from .speech import speech_spans
+ return speech_spans(path, self.model_path(), self.cache(), cancelled)
+
+ def get(self, key):
+ """{'spans': [...]} when known, {'state': 'pending'} while working, {'spans': None, 'reason': …} otherwise."""
+ if not isinstance(key, str) or not re.fullmatch(r'(work|generated|import):[^\x00]{1,512}', key):
+ raise ValueError('Invalid source')
+ try:
+ path = self.audio(key)
+ except (FileNotFoundError, KeyError):
+ return dict(spans=None, reason='source_missing')
+ if path is None:
+ return dict(spans=[])
+ with self.lock:
+ if key in self.known:
+ return dict(spans=self.known[key])
+ model = self.model_path()
+ if not model.is_file():
+ return dict(spans=None, reason='model_missing')
+ from .speech import cached
+ found = cached(path, model, self.cache())
+ if found is not None:
+ return dict(spans=found)
+ with self.lock:
+ if key in self.failed:
+ return dict(spans=None, reason=self.failed[key])
+ if key not in self.pending:
+ if len(self.pending) >= MAX_QUIET:
+ return dict(spans=None, reason='busy')
+ self.pending.append(key)
+ if self.worker is None or not self.worker.is_alive():
+ self.worker = threading.Thread(target=self._run, name='freevideo-editor-speech', daemon=True)
+ self.worker.start()
+ return dict(state='pending')
+
+ def _run(self):
+ from .speech import SpeechError
+ while True:
+ with self.lock:
+ if not self.pending:
+ self.worker = None
+ return
+ key = self.pending[0]
+ reason = found = None
+ try:
+ found = self.compute(self.audio(key))
+ except SpeechError as error:
+ reason = error.reason
+ except Exception: # noqa: BLE001 - one unreadable source must not stop the others
+ logging.getLogger(__name__).warning('Editor speech spans failed for %s.', key, exc_info=True)
+ reason = 'failed'
+ with self.lock:
+ self.pending.popleft()
+ if reason is not None:
+ self.failed[key] = reason
+ else:
+ self.known[key] = found
+
+ def spans(self, keys, cancelled=None):
+ """Speech spans for each key the export needs; a source that cannot be read has none."""
+ found = {}
+ for key in keys:
+ try:
+ path = self.audio(key)
+ found[key] = self.compute(path, cancelled) if path is not None else []
+ except Exception as error: # noqa: BLE001 - no spans for that source, as in the preview
+ if getattr(error, 'reason', None) == 'cancelled':
+ raise
+ logging.getLogger(__name__).warning('Editor export: no speech spans for %s (%s).', key,
+ getattr(error, 'reason', type(error).__name__))
+ return found
+
+
+def register():
+ """Register the editing page's music endpoints once on the existing ComfyUI server."""
+ import asyncio
+ from aiohttp import web
+ from server import PromptServer
+ from .editor_export import _output_root
+ server = PromptServer.instance
+ if server is None or getattr(server, '_freevideo_editor_music', None) is not None:
+ return
+ queue = MusicQueue(_output_root(), server=server)
+ pack = MusicPack()
+ server._freevideo_editor_music = queue
+ server._freevideo_editor_speech = speech = SpeechSpans(_output_root())
+ headers = {'Cache-Control': 'no-store'}
+
+ def response(value):
+ return web.json_response({'job': value}, headers=headers)
+
+ def refused(reason, status):
+ return web.json_response({'reason': reason}, status=status, headers=headers)
+
+ @server.routes.post('/freevideo/editor/music')
+ async def start(request):
+ try:
+ body = await request.json()
+ if not await asyncio.to_thread(available):
+ return refused('unsupported', 409)
+ return response(await asyncio.to_thread(queue.submit, body))
+ except MusicBusy:
+ return refused('busy', 503)
+ except (ValueError, TypeError, KeyError):
+ return refused('invalid_request', 400)
+
+ @server.routes.get('/freevideo/editor/music')
+ async def status(request):
+ identity = request.query.get('id')
+ if identity is None:
+ supported = await asyncio.to_thread(available)
+ try:
+ ready = supported and (await asyncio.to_thread(pack.status))['ready']
+ except Exception: # noqa: BLE001 - an unreadable pack reads as not downloaded
+ ready = False
+ # pace: seconds of composing per second of music here, from the last piece (null before the first).
+ return web.json_response(dict(available=supported, ready=bool(ready), pace=queue.pace), headers=headers)
+ try:
+ return response(queue.get(identity))
+ except KeyError:
+ return refused('missing', 404)
+ except ValueError:
+ return refused('invalid_request', 400)
+
+ @server.routes.post('/freevideo/editor/music/cancel')
+ async def cancel(request):
+ try:
+ body = await request.json()
+ if not isinstance(body, dict) or set(body) != {'id'}:
+ raise ValueError('Invalid request')
+ return response(queue.cancel(body['id']))
+ except KeyError:
+ return refused('missing', 404)
+ except (ValueError, TypeError):
+ return refused('invalid_request', 400)
+
+ @server.routes.get('/freevideo/editor/speech')
+ async def speech_spans(request):
+ try:
+ return web.json_response(await asyncio.to_thread(speech.get, request.query.get('source')), headers=headers)
+ except ValueError:
+ return refused('invalid_request', 400)
+
+ @server.routes.get('/freevideo/editor/music-pack')
+ async def pack_status(request):
+ return web.json_response(await asyncio.to_thread(pack.status), headers=headers)
+
+ @server.routes.post('/freevideo/editor/music-pack')
+ async def pack_action(request):
+ try:
+ body = await request.json()
+ if not isinstance(body, dict) or set(body) != {'action'} or body['action'] not in ('install', 'cancel'):
+ raise ValueError('Invalid request')
+ if body['action'] == 'cancel':
+ pack.cancel()
+ elif not await asyncio.to_thread(available):
+ return refused('unsupported', 409)
+ elif not await asyncio.to_thread(pack.start):
+ return web.json_response(await asyncio.to_thread(pack.status), status=507, headers=headers)
+ except (ValueError, TypeError):
+ return refused('invalid_request', 400)
+ return web.json_response(await asyncio.to_thread(pack.status), headers=headers)
diff --git a/freevideo_engine/image_gen/image_models.json b/freevideo_engine/image_gen/image_models.json
index 797515d..86f7f86 100644
--- a/freevideo_engine/image_gen/image_models.json
+++ b/freevideo_engine/image_gen/image_models.json
@@ -661,6 +661,29 @@
"url": "https://www.modelscope.cn/models/ACE-Step/Ace-Step1.5/resolve/dd20ee00556360ae895e308efe9d11daf08fe9d2/vae/diffusion_pytorch_model.safetensors"
}
]
+ },
+ {
+ "file": "music/silero-vad/model.onnx",
+ "component": "music",
+ "bytes": 2243022,
+ "sha256": "a4a068cd6cf1ea8355b84327595838ca748ec29a25bc91fc82e6c299ccdc5808",
+ "repo": "onnx-community/silero-vad",
+ "revision": "e71cae966052b992a7eca6b17738916ce0eca4ec",
+ "model_id": "silero-vad",
+ "sources": [
+ {
+ "id": "official",
+ "url": "https://huggingface.co/onnx-community/silero-vad/resolve/e71cae966052b992a7eca6b17738916ce0eca4ec/onnx/model.onnx"
+ },
+ {
+ "id": "hf-mirror",
+ "url": "https://hf-mirror.com/onnx-community/silero-vad/resolve/e71cae966052b992a7eca6b17738916ce0eca4ec/onnx/model.onnx"
+ },
+ {
+ "id": "modelscope",
+ "url": "https://www.modelscope.cn/models/onnx-community/silero-vad/resolve/2f080fcc38651e65b738157ad84c3c980beb08cc/onnx/model.onnx"
+ }
+ ]
}
]
}
diff --git a/freevideo_engine/image_gen/image_pack.py b/freevideo_engine/image_gen/image_pack.py
index fe60e97..4b252f0 100644
--- a/freevideo_engine/image_gen/image_pack.py
+++ b/freevideo_engine/image_gen/image_pack.py
@@ -229,7 +229,9 @@ SEGMENT_NOTICE_TEXT = (b'SAM 2.1 Hiera small, Copyright (c) Meta Platforms, Inc.
b'https://github.com/facebookresearch/sam2\n')
MUSIC_NOTICE_TEXT = (b'ACE-Step 1.5, Copyright (c) 2026 ACEStep. Licensed under the MIT License; see '
b'freevideo_engine/licenses/ACE-Step-MIT.txt.\nhttps://github.com/ace-step/ACE-Step-1.5\n'
- b'Qwen3-Embedding-0.6B, Copyright (c) Alibaba Cloud. Licensed under Apache-2.0.\n')
+ b'Qwen3-Embedding-0.6B, Copyright (c) Alibaba Cloud. Licensed under Apache-2.0.\n'
+ b'Silero VAD, Copyright (c) 2020-present Silero Team. Licensed under the MIT License; see '
+ b'freevideo_engine/licenses/Silero-VAD-MIT.txt.\nhttps://github.com/snakers4/silero-vad\n')
NOTICES = {'image': ('NOTICE.txt', NOTICE_TEXT), 'segment': ('SAM2-NOTICE.txt', SEGMENT_NOTICE_TEXT),
'music': ('MUSIC-NOTICE.txt', MUSIC_NOTICE_TEXT)}
diff --git a/freevideo_engine/speech.py b/freevideo_engine/speech.py
new file mode 100644
index 0000000..0a8fd04
--- /dev/null
+++ b/freevideo_engine/speech.py
@@ -0,0 +1,279 @@
+"""Where someone speaks in a sound file, so generated music can sit lower under dialogue.
+
+Silero VAD (snakers4/silero-vad, MIT; the onnx-community export) runs on the CPU in torch: the 16 kHz
+weights are read straight from the ONNX file, so no ONNX runtime is needed. The file is downloaded with the
+music model. One pass over a file gives speech spans in seconds of that file; they are cached next to the
+editor's other derived data, and the preview and the export read the same spans.
+"""
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+from pathlib import Path
+import threading
+
+import numpy as np
+
+MODEL_FILE = 'music/silero-vad/model.onnx'
+VERSION = 1 # change when the spans for the same file would change
+RATE = 16000
+WINDOW, CONTEXT = 512, 64
+ENTER, LEAVE = .5, .35 # Silero's own hysteresis
+MIN_SPEECH, MIN_SILENCE, PAD = .25, .1, .03
+# Music under dialogue stays down across short pauses between sentences instead of pumping.
+JOIN_GAP = .6
+LIMIT_SECONDS = 4 * 3600
+
+
+class SpeechError(Exception):
+ def __init__(self, reason):
+ super().__init__(reason)
+ self.reason = reason
+
+
+# --- a small protobuf reader: only what an ONNX file's tensors need -----------------------------------------
+def _varint(data, index):
+ shift = value = 0
+ while True:
+ byte = data[index]
+ index += 1
+ value |= (byte & 0x7f) << shift
+ if byte < 0x80:
+ return value, index
+ shift += 7
+
+
+def _fields(data):
+ index = 0
+ while index < len(data):
+ key, index = _varint(data, index)
+ number, wire = key >> 3, key & 7
+ if wire == 0:
+ value, index = _varint(data, index)
+ elif wire == 1:
+ value, index = data[index:index + 8], index + 8
+ elif wire == 2:
+ size, index = _varint(data, index)
+ value, index = data[index:index + size], index + size
+ elif wire == 5:
+ value, index = data[index:index + 4], index + 4
+ else:
+ raise SpeechError('model')
+ yield number, wire, value
+
+
+def _tensor(data):
+ dims, kind, name, raw = [], None, '', None
+ for number, wire, value in _fields(data):
+ if number == 1:
+ if wire == 2:
+ index = 0
+ while index < len(value):
+ dim, index = _varint(value, index)
+ dims.append(dim)
+ else:
+ dims.append(value)
+ elif number == 2:
+ kind = value
+ elif number == 8:
+ name = bytes(value).decode()
+ elif number == 9:
+ raw = value
+ if kind != 1 or raw is None or len(raw) != 4 * int(np.prod(dims or [1])):
+ return name, None
+ return name, np.frombuffer(bytes(raw), '<f4').reshape(dims).copy()
+
+
+def _tensors(graph, found):
+ for number, _, value in _fields(graph):
+ if number == 5:
+ name, array = _tensor(value)
+ if array is not None:
+ found[name] = array
+ elif number == 1:
+ for field, _, attribute in _fields(value):
+ if field == 5:
+ for part, _, inner in _fields(attribute):
+ if part == 6:
+ _tensors(inner, found)
+ return found
+
+
+def onnx_weights(path):
+ """The 16 kHz branch of the onnx-community Silero export, as numpy arrays named for _Model."""
+ data = memoryview(Path(path).read_bytes())
+ found = {}
+ for number, _, value in _fields(data):
+ if number == 7:
+ _tensors(value, found)
+ prefix = 'If_0_then_branch__Inline_0__'
+ lstm = [array for name, array in found.items()
+ if name.startswith(prefix) and name.endswith('_subg_96_sub_graph2') and array.ndim >= 2]
+ try:
+ weights = {'basis': found[prefix + 'stft.forward_basis_buffer']}
+ for layer in range(4):
+ weights[f'conv{layer}.weight'] = found[f'{prefix}encoder.{layer}.reparam_conv.weight']
+ weights[f'conv{layer}.bias'] = found[f'{prefix}encoder.{layer}.reparam_conv.bias']
+ weights['out.weight'] = found[prefix + 'decoder.decoder.2.weight']
+ weights['out.bias'] = found[prefix + 'decoder.decoder.2.bias']
+ w, r = [array[0] for array in lstm if array.shape == (1, 512, 128)]
+ (b,) = [array[0] for array in lstm if array.shape == (1, 1024)]
+ except (KeyError, ValueError):
+ raise SpeechError('model') from None
+ # ONNX gates are input, output, forget, cell; torch's are input, forget, cell, output.
+ order = lambda x: np.concatenate([x[0:128], x[256:384], x[384:512], x[128:256]])
+ weights.update({'lstm.weight_ih_l0': order(w), 'lstm.weight_hh_l0': order(r),
+ 'lstm.bias_ih_l0': order(b[:512]), 'lstm.bias_hh_l0': order(b[512:])})
+ if weights['basis'].shape != (258, 1, 256) or weights['conv0.weight'].shape != (128, 129, 3):
+ raise SpeechError('model')
+ return weights
+
+
+def _model(weights):
+ import torch
+ from torch import nn
+
+ class Model(nn.Module):
+ def __init__(self):
+ super().__init__()
+ self.register_buffer('basis', torch.zeros(258, 1, 256))
+ self.conv0 = nn.Conv1d(129, 128, 3, padding=1)
+ self.conv1 = nn.Conv1d(128, 64, 3, stride=2, padding=1)
+ self.conv2 = nn.Conv1d(64, 64, 3, stride=2, padding=1)
+ self.conv3 = nn.Conv1d(64, 128, 3, padding=1)
+ self.lstm = nn.LSTM(128, 128, batch_first=True)
+ self.out = nn.Conv1d(128, 1, 1)
+
+ def forward(self, windows):
+ """[n, 576] windows (64 context + 512 new samples) in order → [n] speech probabilities."""
+ x = nn.functional.pad(windows, (0, 64), mode='reflect').unsqueeze(1)
+ x = nn.functional.conv1d(x, self.basis, stride=128)
+ x = torch.sqrt(x[:, :129] ** 2 + x[:, 129:] ** 2)
+ for conv in (self.conv0, self.conv1, self.conv2, self.conv3):
+ x = torch.relu(conv(x))
+ x, _ = self.lstm(x.squeeze(-1).unsqueeze(0))
+ x = torch.sigmoid(self.out(torch.relu(x[0]).unsqueeze(-1)))
+ return x[:, 0, 0]
+
+ model = Model()
+ model.load_state_dict({name: torch.from_numpy(array) for name, array in weights.items()})
+ return model.eval()
+
+
+_LOADED = {}
+_LOCK = threading.Lock()
+
+
+def probabilities(samples, model_path):
+ """float32 mono samples at 16 kHz → one speech probability per 512 samples (32 ms)."""
+ import torch
+ with _LOCK:
+ key = str(model_path)
+ if key not in _LOADED:
+ _LOADED.clear()
+ _LOADED[key] = _model(onnx_weights(model_path))
+ model = _LOADED[key]
+ samples = np.asarray(samples, np.float32)
+ count = -(-len(samples) // WINDOW)
+ if count == 0:
+ return np.zeros(0, np.float32)
+ padded = np.zeros(CONTEXT + count * WINDOW, np.float32)
+ padded[CONTEXT:CONTEXT + len(samples)] = samples
+ windows = np.lib.stride_tricks.sliding_window_view(padded, CONTEXT + WINDOW)[::WINDOW][:count]
+ threads = torch.get_num_threads()
+ try:
+ torch.set_num_threads(min(threads, 4))
+ with torch.inference_mode():
+ return model(torch.from_numpy(np.ascontiguousarray(windows))).numpy()
+ finally:
+ torch.set_num_threads(threads)
+
+
+def spans(probs, duration):
+ """Speech probabilities → merged [start, end] seconds, with Silero's hysteresis and padding."""
+ step = WINDOW / RATE
+ found, start, quiet = [], None, None
+ for index, value in enumerate(probs):
+ at = index * step
+ if start is None:
+ if value >= ENTER:
+ start, quiet = at, None
+ elif value < LEAVE:
+ quiet = at if quiet is None else quiet
+ if at + step - quiet >= MIN_SILENCE:
+ found.append([start, quiet])
+ start, quiet = None, None
+ else:
+ quiet = None
+ if start is not None:
+ found.append([start, len(probs) * step])
+ joined = []
+ for begin, end in found:
+ if end - begin < MIN_SPEECH:
+ continue
+ begin, end = max(0., begin - PAD), min(duration, end + PAD)
+ if joined and begin - joined[-1][1] <= JOIN_GAP:
+ joined[-1][1] = max(joined[-1][1], end)
+ else:
+ joined.append([begin, end])
+ return [[round(begin, 3), round(end, 3)] for begin, end in joined]
+
+
+def decode(path, cancelled=None):
+ """The first audio stream of a media file as float32 mono at 16 kHz."""
+ import av
+ parts, total = [], 0
+ try:
+ with av.open(str(path)) as container:
+ if not container.streams.audio:
+ return np.zeros(0, np.float32)
+ stream = container.streams.audio[0]
+ resampler = av.AudioResampler(format='flt', layout='mono', rate=RATE)
+ for frame in container.decode(stream):
+ for out in resampler.resample(frame):
+ part = out.to_ndarray().reshape(-1)
+ parts.append(part)
+ total += len(part)
+ if total > LIMIT_SECONDS * RATE:
+ raise SpeechError('too_long')
+ if cancelled is not None and cancelled():
+ raise SpeechError('cancelled')
+ for out in resampler.resample(None):
+ parts.append(out.to_ndarray().reshape(-1))
+ except av.error.FFmpegError:
+ raise SpeechError('unreadable') from None
+ return np.concatenate(parts).astype(np.float32) if parts else np.zeros(0, np.float32)
+
+
+def _identity(path, model_path):
+ stat, model = Path(path).stat(), Path(model_path).stat()
+ raw = json.dumps([str(Path(path).resolve()), stat.st_size, stat.st_mtime_ns, model.st_size, VERSION])
+ return hashlib.sha256(raw.encode()).hexdigest()[:32]
+
+
+def cached(path, model_path, cache):
+ """The cached spans for this exact file and model, or None."""
+ try:
+ value = json.loads((Path(cache) / (_identity(path, model_path) + '.json')).read_text(encoding='utf-8'))
+ return value['spans'] if isinstance(value, dict) and isinstance(value.get('spans'), list) else None
+ except (OSError, ValueError, KeyError):
+ return None
+
+
+def speech_spans(path, model_path, cache, cancelled=None):
+ """Speech spans in seconds of `path`; computed once per file and model, then read from `cache`."""
+ found = cached(path, model_path, cache)
+ if found is not None:
+ return found
+ if not Path(model_path).is_file():
+ raise SpeechError('model_missing')
+ samples = decode(path, cancelled)
+ found = spans(probabilities(samples, model_path), len(samples) / RATE)
+ folder = Path(cache)
+ folder.mkdir(parents=True, exist_ok=True)
+ target = folder / (_identity(path, model_path) + '.json')
+ temporary = target.with_name(f'{target.name}.{os.getpid()}.{threading.get_ident()}.tmp')
+ temporary.write_text(json.dumps({'spans': found}), encoding='utf-8')
+ os.replace(temporary, target)
+ return found
diff --git a/web/editor.css b/web/editor.css
index 8375feb..a6ea01d 100644
--- a/web/editor.css
+++ b/web/editor.css
@@ -84,6 +84,14 @@
.fv-editor .fv-ed-gen-retry:disabled{color:var(--fv-muted);cursor:default}
.fv-ed-genbar .fv-request-assets{order:-2;flex:1 1 100%;min-width:0;font-size:12px;line-height:1.5}
.fv-ed-genbar .fv-request-assets[hidden]{display:none}
+/* A stretch of the music track: a style row and a mood row above the description. */
+.fv-ed-genbar.fv-ed-genbar-music{flex-wrap:wrap}
+.fv-ed-music-presets{order:-1;flex:1 1 100%;display:flex;flex-direction:column;gap:6px;min-width:0}
+.fv-ed-music-presets[hidden]{display:none}
+.fv-ed-music-row{display:flex;align-items:center;gap:8px;min-width:0}
+.fv-ed-music-label{flex:none;min-width:2.5em;font-size:13px;color:var(--fv-muted)}
+.fv-editor .fv-ed-music-presets .fv-ed-adjust-segmented{width:auto;flex:0 1 auto;min-width:0;flex-wrap:wrap}
+.fv-editor .fv-ed-music-presets .fv-ed-adjust-segmented button{flex:none}
.fv-editor:has(.fv-ed-gen-failure:not([hidden])){--fv-ed-generation-row:max-content}
.fv-ed-genbar:has(.fv-ed-gen-failure:not([hidden])){flex-wrap:wrap;padding-top:10px;padding-bottom:10px}
.fv-editor .fv-ed-gen-failure{display:block;flex:1 0 100%;min-width:0}
diff --git a/web/editor.js b/web/editor.js
index d1d3103..954043f 100644
--- a/web/editor.js
+++ b/web/editor.js
@@ -21,9 +21,11 @@ import {createAdjustPicture} from './editor_adjust_picture.js';
import {pictureSession} from './editor_picture_session.js';
import {createPictureTools} from './editor_frame.js';
import {openExportDialog} from './editor_export_dialog.js';
-import {createAudioIntegration, handleAudioShortcut, AUDIO_SHORTCUTS} from './editor_audio.js';
+import {createAudioIntegration, createSpeechSpans, handleAudioShortcut, AUDIO_SHORTCUTS} from './editor_audio.js';
import {createSpeedComponent, readSpeedAudioSamples, handleSpeedKey, speedShortcuts} from './editor_speed.js';
import {createEditorGenbar, loadEditorComponents, editorBlockStates, blockMessages} from './editor_genbar.js';
+import {createEditorMusic} from './editor_music.js';
+import {MAX_MUSIC_TEXT as MUSIC_TEXT_LIMIT} from './editor_music_model.js';
import {createProgressConnection} from './progress_connection.js';
import {createSelectionActions, deletionStatus, isMultiple, isTyping, playheadIds, removableGaps, removeGaps} from './editor_select.js';
import {isolateDialogKeys, topDialog} from './dialog_keys.js';
@@ -188,8 +190,9 @@ export function openEditor({api, app, from = null, panel = {}, components: given
const t = (en, zh) => String(language()).toLowerCase().startsWith('zh') ? zh : en;
const returnFocus = document.activeElement;
const even = value => Math.max(2, Math.floor(value / 2) * 2);
- const initial = () => model.emptyProject({width: even(panel.width || 1344), height: even(panel.height || 768), settings: panel.settings || null,
- name: t('Untitled edit', '未命名剪辑')});
+ // A new edit opens with an empty second sound track, where music is made.
+ const initial = () => model.addAudioTrack(model.emptyProject({width: even(panel.width || 1344), height: even(panel.height || 768), settings: panel.settings || null,
+ name: t('Untitled edit', '未命名剪辑')}));
const state = createEditorState(initial());
const resumeCanvasPointer = suspendCanvasPointer(app?.canvas);
const element = el('dialog', 'fv-studio fv-editor'); element.setAttribute('aria-label', t('Edit', '剪辑视图')); element.tabIndex = -1;
@@ -374,7 +377,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given
}, {onStatus: () => {
if (!queue.dirty) { discardOnSwitch = false; saveWarning.hidden = true; }
}});
- const player = createEditorPlayer(preview, transport, {state, sources: store.sources, api, t, error, readAudioSamples: panel.readAudioSamples || readSpeedAudioSamples, preservePreview: () => initializing, prepareAudio: () => !initializing});
+ const player = createEditorPlayer(preview, transport, {state, sources: store.sources, api, t, error, readAudioSamples: panel.readAudioSamples || readSpeedAudioSamples, preservePreview: () => initializing, prepareAudio: () => !initializing, speech: createSpeechSpans(store.speech)});
function selectionClips(selection = state.selection) {
if (selection?.kind !== 'clip') return [];
return state.record.tracks[0].clips.filter(clip => selection.ids.includes(clip.id) || selection.ids.includes(clip.link));
@@ -515,6 +518,9 @@ export function openEditor({api, app, from = null, panel = {}, components: given
function completeSources() {
return store.getSources(model.references(state.record).map(sourceKey)).then(() => { player.refresh(); tracks?.refresh?.(); });
}
+ // Music on the second sound track goes through the same bar; the bar shows a chosen piece's own job.
+ const music = createEditorMusic({state, store, t, completeSources, onStatus: () => genbar?.update(barValue)});
+ let barValue = null;
const exportMatches = () => exportSnapshot && exportSnapshot.id === state.record.id && exportSnapshot.revision === state.record.revision && exportSnapshot.content === content(state.record);
function explainExport() {
if (initializing || switching) { notify(t('Opening the edit…', '正在打开…')); return true; }
@@ -655,7 +661,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given
initialSelection = offerInitial = extendEntry = null;
projectEpoch++; held.length = 0;
player.pause(); for (const stop of jobWatchers.values()) stop(); jobWatchers.clear(); jobStatuses.clear(); progress.clear(); runProgress.clear(); failures.clear(); progressConnection?.reset();
- mostRecentJob = null; submissionFailure = null; genbar?.update(null); queue.reset(record); stopExport?.(); stopExport = null; exportStatus = exportSnapshot = null; exportButton.textContent = t('Export', '导出');
+ mostRecentJob = null; submissionFailure = null; music.reset(); genbar?.update(null); queue.reset(record); stopExport?.(); stopExport = null; exportStatus = exportSnapshot = null; exportButton.textContent = t('Export', '导出');
suppressSave = true; try { state.replace(record); savedContent = content(record); } finally { suppressSave = false; }
panel.onProjectLoaded?.(record);
const width = tracksHost.clientWidth - 148; if (width > 0) state.setZoom(model.openingZoom(width, record));
@@ -689,7 +695,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given
forgetPointer();
})); if (disposed) return;
// A restored edit remains usable offline with its existing placeholders.
- await completeSources().catch(value => error(value, null, undefined, 'sources')); if (disposed) return; recoverJobs();
+ await completeSources().catch(value => error(value, null, undefined, 'sources')); if (disposed) return; recoverJobs(); music.recover();
state.restoreView(remembered);
// Also when generating is confirmed only after the edit has opened.
offerInitial = () => { if (epoch === projectEpoch && !state.selection && state.record.tracks.every(track => !track.clips.length)) selectInitial(); };
@@ -844,6 +850,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given
interrupted: clip => clip.pending === 'stopped' && !clip.source && !!clip.job
&& (jobStatuses.get(clip.job)?.blocks || []).some(block => block.state === 'interrupted' && model.attemptClip(state.record, block.id)?.id === clip.id),
selectedFailure() {
+ const tune = music.failure(); if (tune) return tune;
const clip = state.selection?.kind === 'clip' && state.record.tracks.flatMap(track => track.clips).find(clip => state.selection.ids.includes(clip.id) && clip.pending === 'failed');
if (clip) return {id: clip.id, text: failures.text(clip.id)};
return submissionFailure === JSON.stringify(state.selection) ? {id: 'submission', text: failures.text('submission')} : null;
@@ -875,11 +882,13 @@ export function openEditor({api, app, from = null, panel = {}, components: given
// The description the box shows for a choice of planned parts, and the parts it is written to; null for
// any other choice.
partText(selection = state.selection) {
+ if (music.target(selection)) return music.text(selection);
const clips = selectionClips(selection).sort((a, b) => a.start - b.start);
return clips.length && clips.every(editablePart) ? {ids: clips.map(clip => clip.id), text: clips[0].text} : null;
},
describeParts(ids, text) {
text = wellFormed(text);
+ if (ids.length === 1 && music.target({kind: 'clip', ids})) return music.describe(ids[0], text);
if (!ids.every(id => editablePart(model.find(state.record, id)))) return false;
try { state.apply(t('Edit description', '修改说明'), record => model.describePlanned(record, ids, text)); return true; }
catch (value) { if (value instanceof model.EditError) return false; throw value; }
@@ -915,6 +924,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given
}},
previewRange: selection => genbar?.preview(selection),
async estimate(selection) {
+ if (music.target(selection)) return music.estimate(selection);
const run = partRun(selection), range = selectionRange(run ? clipChoice(run) : selection); if (!range) return null;
let seconds = 0;
for (const count of model.rangeParts(state.record, {...range, ...capability})) { const value = capability.estimate(count); if (value == null) return null; seconds += value; }
@@ -1012,6 +1022,11 @@ export function openEditor({api, app, from = null, panel = {}, components: given
async press(text, {allowEmpty = false} = {}) {
if (disposed || closing || initializing) return {say: t('Opening the edit…', '正在打开…')};
text = wellFormed(text);
+ // A stretch of the music track, or a piece of music there: music, with or without words.
+ if (music.target()) {
+ if (characters(text) > MUSIC_TEXT_LIMIT) return {say: textLongText(t, MUSIC_TEXT_LIMIT), focus: true};
+ return music.press(text);
+ }
let selection = state.selection, said = null;
if (actions.selectionHeld()) return {say: t('Queued', '排队中')};
if (busySelection(selection)) return {say: t('This part is generating.', '这一段正在生成。')};
@@ -1076,9 +1091,9 @@ export function openEditor({api, app, from = null, panel = {}, components: given
await actions.generate({text: texts ? texts[run[0].id] : text, texts, selection, shown, allowEmpty});
return {text, say: [said, used, busyBefore ? t('Queued. It starts after the part now generating.', '已排队,正在生成的那段完成后自动开始。') : null].filter(Boolean).join(' ') || null, queued: busyBefore};
},
- selectionRunning: () => busySelection(state.selection),
+ selectionRunning: () => music.target() ? music.running() : busySelection(state.selection),
// A choice over existing clips that Generate would move to an empty spot.
- selectionMovable: () => !actions.supportedSelection() && !!emptySpot(state.selection),
+ selectionMovable: () => !music.target() && !actions.supportedSelection() && !!emptySpot(state.selection),
selectionSending() { const range = selectionRange(state.selection); return !!range && ([...sendingRanges].some(other => overlaps(other, range)) || [...sending].some(id => { const clip = model.find(state.record, id); return clip && overlaps(clip, range); })); },
selectionHeld() { const ids = selectionClips(state.selection).map(clip => clip.id); return ids.length > 0 && ids.every(isHeld); },
// Takes the selected parts out of the line; they stay as drafts.
@@ -1088,6 +1103,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given
rememberView();
},
async cancel() {
+ if (music.target()) { await music.cancel(); return; }
const range = selectionRange(state.selection);
const own = range && state.record.tracks[0].clips.find(clip => clip.job && generating(clip)
&& clip.start < range.start + range.frames && clip.start + clip.frames > range.start)?.job;
@@ -1214,7 +1230,16 @@ export function openEditor({api, app, from = null, panel = {}, components: given
return running;
}
+ actions.music = {target: selection => music.target(selection), preset: selection => music.preset(selection), choose: value => music.choose(value),
+ styles: music.styles, moods: music.moods, downloadBytes: music.downloadBytes};
genbar = createEditorGenbar(genHost, {state, t, actions, text: ''}); settleService();
+ {
+ // What the editor gives the bar is the video job's; a chosen piece of music being made shows its own instead.
+ const update = genbar.update;
+ genbar.update = value => { barValue = value; update(music.barStatus() || value); };
+ state.subscribe(event => { if (event.type === 'selection' || event.type === 'record') update(music.barStatus() || barValue); });
+ music.refreshInfo();
+ }
const adjust = adjustPanel = createAdjustPanel(adjustHost, {state, t, actions});
const pictureEdit = pictureSession(state, t);
const notes = createBlockNotes({state, t, actions}); actions.blockNote = notes.decorate; actions.describedTrack = describedTrack;
@@ -1315,7 +1340,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given
const next = content(state.record);
if (!suppressSave && !initializing && next !== savedContent && exportStatus?.state === 'done') { exportStatus = null; exportButton.textContent = t('Export', '导出'); }
if (!suppressSave && !initializing && next !== savedContent) { savedContent = next; queue.enqueue(state.record); }
- completeSources().catch(value => error(value, null, undefined, 'sources')); recoverJobs();
+ completeSources().catch(value => error(value, null, undefined, 'sources')); recoverJobs(); music.recover();
}
if (event.type === 'selection') { if (state.selection !== initialSelection) initialSelection = null; }
if (event.type === 'record' || event.type === 'selection') { updateTop(); updateTransport(); pictureTools.update(); }
@@ -1413,7 +1438,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given
// A workflow without FreeVideo nodes came to the front: the editing view makes way (unsaved edits stay open).
const collapse = () => { void handle.close(); };
window.addEventListener('freevideo-collapse', collapse);
- const handle = {element, state, actions, store, adjust, audio, notify,
+ const handle = {element, state, actions, store, adjust, audio, music, notify,
async switchView(view) {
if (closing || disposed || view === 'editor') return;
if (await handle.close() !== false) await onView(view);
@@ -1440,7 +1465,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given
}
}
await store.activateProject(null).catch(() => {});
- rememberView(); disposed = true; guide?.dispose(); transportControls.dispose(); toolbarMore.dispose(); player.dispose();
+ rememberView(); disposed = true; music.dispose(); guide?.dispose(); transportControls.dispose(); toolbarMore.dispose(); player.dispose();
textTools.dispose(); adjust.dispose(); notes.dispose(); pictureEdit.dispose();
audio.dispose(); headerFit.dispose();
listeners.splice(0).forEach(remove => remove());
@@ -1472,8 +1497,8 @@ export function openEditor({api, app, from = null, panel = {}, components: given
record = await store.byWork(id);
if (!record) {
const key = `work:${id}`; await store.getSources([key]); const metadata = store.sources.get(key), g = work.geometry || {};
- record = model.emptyProject({width: even(metadata?.width || g.width || panel.width || 1344), height: even(metadata?.height || g.height || panel.height || 768),
- settings: panel.settings || null, name: t('Untitled edit', '未命名剪辑')});
+ record = model.addAudioTrack(model.emptyProject({width: even(metadata?.width || g.width || panel.width || 1344), height: even(metadata?.height || g.height || panel.height || 768),
+ settings: panel.settings || null, name: t('Untitled edit', '未命名剪辑')}));
record = model.insertSource(record, {source: {kind: 'work', id}, frames: metadata?.frames || g.frames || Math.round((g.seconds || 15) * 24),
hasVideo: true, hasAudio: metadata?.has_audio ?? work.has_audio !== false, text: metadata?.text, at: 0});
if (!metadata) store.cache({[key]: {video: mediaURL(api, work.video), audio: mediaURL(api, work.video), frames: g.frames,
diff --git a/web/editor_audio.js b/web/editor_audio.js
index b5b0fb3..40c79ac 100644
--- a/web/editor_audio.js
+++ b/web/editor_audio.js
@@ -1,5 +1,5 @@
import * as model from './editor_model.js';
-import {row, slider, number, disabledReason} from './editor_adjust.js';
+import {row, slider, number, toggle, disabledReason} from './editor_adjust.js';
import * as sound from './editor_audio_model.js';
import * as rules from './editor_audio_rules.js';
import {replaceMode, attachReplacePicker} from './editor_audio_replace.js';
@@ -127,6 +127,11 @@ export function createAudioPanel(host, {state, player, t = en => en, error = ()
onChange: gain => { const ids = selectedAudio(state).map(clip => clip.id); editingVolume = false;
apply(t('Volume', '音量'), record => ids.reduce((next, id) => model.setVolume(next, id, gain / 100), record)); },
});
+ // Generated music only: lowered while someone speaks in the other sound. On by default.
+ const duck = toggle(t('Lower during speech', '对白时降低音量'), {
+ onChange: on => apply(t('Lower during speech', '对白时降低音量'), record => sound.setDuck(record, selectedAudio(state).filter(sound.duckable).map(clip => clip.id), on)),
+ });
+ duck.classList.add('fv-ed-audio-duck'); form.append(duck); controls.push(duck);
const fadeIn = input(form, 'fade-in', t('Fade in', '淡入'), 0, 5, 1 / 24, 'number', {
onChange: seconds => apply(t('Fade in', '淡入'), record => sound.setFades(record, selectedAudio(state).map(clip => clip.id), 'fade_in', Math.round(seconds * 24))),
});
@@ -177,6 +182,9 @@ export function createAudioPanel(host, {state, player, t = en => en, error = ()
const locked = selected.some(clip => sound.trackOf(state.record, clip.id).locked);
const reason = disabledReason(ctx, {needs: 'audio'}) || (locked ? t('This track is locked', '这条轨道已锁定') : '');
for (const field of [volume, fadeIn, fadeOut]) { field.disabled = !!reason; field.title = ''; field.input.title = ''; }
+ const music = clips.filter(sound.duckable);
+ duck.hidden = !music.length || music.length !== clips.length; duck.disabled = !!reason;
+ if (!duck.hidden) { duck.value = music.every(clip => !!clip.duck); duck.input.indeterminate = music.some(clip => !!clip.duck) && !duck.value; }
if (!editingVolume) {
const gains = clips.map(clip => Math.round((clip.volume ?? 1) * 100)), mixed = gains.length > 1 && gains.some(gain => gain !== gains[0]);
volume.value = mixed ? 100 : gains[0] ?? 100; setMixed(volume, mixed);
@@ -314,10 +322,27 @@ export function attachAudioTracks(element, {state, t = en => en, actions = {}})
// A decoded, bounded look-ahead path gives the sample envelope to Web Audio.
// The unchanged media path remains active for records without audio edits.
-export function createAudioPreview({state, sources, api, volumePreview = new Map(), readSamples = null, error = () => {}}) {
+// Speech spans per source for "Lower during speech": asked once, polled while the server works them out.
+// A source that cannot be read, or a server without the route, has no speech: nothing is lowered under it.
+export function createSpeechSpans(fetchSpans, {retry = 1000} = {}) {
+ const map = new Map(), asking = new Set(); let version = 0, disposed = false;
+ const ask = key => {
+ if (disposed || map.has(key) || asking.has(key)) return;
+ asking.add(key);
+ Promise.resolve().then(() => fetchSpans(key)).then(value => {
+ asking.delete(key); if (disposed) return;
+ if (value?.state === 'pending' || value?.reason === 'model_missing') { setTimeout(() => ask(key), value?.state === 'pending' ? retry : retry * 10); return; }
+ map.set(key, Array.isArray(value?.spans) ? value.spans : []); version++;
+ }).catch(() => { asking.delete(key); if (!disposed) { map.set(key, []); version++; } });
+ };
+ return {map, want(keys) { keys.forEach(ask); }, get version() { return version; }, dispose() { disposed = true; }};
+}
+export function createAudioPreview({state, sources, api, volumePreview = new Map(), readSamples = null, error = () => {}, speech = null}) {
let context = null, generation = 0, timer = null, identity = null, pending = false, disposed = false;
const cache = new Map(), nodes = new Set(); let origin = 0, cursor = 0, previousPosition = 0;
- const edited = record => record.tracks.some(track => track.muted || track.solo || track.clips.some(clip => clip.fade_in || clip.fade_out || clip.crossfade));
+ const ducking = (record, rate = 48000) => speech && sound.duckSources(record.tracks).length ? sound.duckRanges(record.tracks, speech.map, rate) : [];
+ const edited = record => record.tracks.some(track => track.muted || track.solo || track.clips.some(clip => clip.fade_in || clip.fade_out || clip.crossfade))
+ || ducking(record).length > 0;
function stop() { generation++; clearTimeout(timer); timer = null; pending = false; identity = null; nodes.forEach(node => { try { node.stop(); } catch {} }); nodes.clear(); }
async function decoded(clip) {
const metadata = sources.get(sourceKey(clip.source)), url = mediaURL(api, metadata?.audio);
@@ -343,7 +368,7 @@ export function createAudioPreview({state, sources, api, volumePreview = new Map
return raw(await decoded(clip), clip, offset, count);
}
async function chunk(record, at, count) {
- const rate = context.sampleRate, total = [new Float64Array(count), new Float64Array(count)];
+ const rate = context.sampleRate, total = [new Float64Array(count), new Float64Array(count)], lowered = ducking(record, rate);
for (const track of record.tracks.filter(track => track.kind === 'audio')) {
if (!sound.trackAudible(track, record.tracks)) continue;
const block = [new Float64Array(count), new Float64Array(count)], extras = {};
@@ -353,6 +378,7 @@ export function createAudioPreview({state, sources, api, volumePreview = new Map
const a = Math.max(start, at), b = Math.min(stop, at + count);
if (b <= a) continue;
const data = await prepared(clip, a - start, b - a);
+ const duck = clip.duck && lowered.length ? sound.duckGain(lowered, a, b - a, rate) : null;
data.forEach((channel, c) => { for (let i = 0; i < channel.length; i++) {
const local = a - start + i, length = stop - start;
const prior = track.clips[track.clips.indexOf(clip) - 1], next = track.clips[track.clips.indexOf(clip) + 1];
@@ -360,6 +386,7 @@ export function createAudioPreview({state, sources, api, volumePreview = new Map
let gain = clip.volume ?? 1;
if (!clip.fade_in && (!prior?.source || prior.start + prior.frames !== clip.start || !prior.volume)) gain *= Math.min(1, local / (edge - 1));
if (!clip.fade_out && (!next?.source || next.start !== clip.start + clip.frames || !next.volume)) gain *= Math.min(1, (length - 1 - local) / (edge - 1));
+ if (duck) gain *= duck[i];
block[c][a - at + i] = channel[i] * gain;
}});
}
@@ -396,6 +423,7 @@ export function createAudioPreview({state, sources, api, volumePreview = new Map
}
return {
render({record, position, playing, playbackRate = 1}) {
+ speech?.want(sound.duckSources(record.tracks));
if (!edited(record)) { if (identity) stop(); return false; }
if (volumePreview.size) record = {...record, tracks: record.tracks.map(track => ({...track, clips: track.clips.map(clip => volumePreview.has(clip.id) ? {...clip, volume: volumePreview.get(clip.id)} : clip)}))};
if (!playing) { stop(); previousPosition = position; return true; }
@@ -404,7 +432,7 @@ export function createAudioPreview({state, sources, api, volumePreview = new Map
if (identity !== unavailable) { stop(); identity = unavailable; error(new model.EditError('audio_prepare')); }
return true;
}
- const nextIdentity = JSON.stringify(record.tracks);
+ const nextIdentity = JSON.stringify(record.tracks) + `:${speech?.version ?? 0}`;
if (identity !== nextIdentity || Math.abs(position - previousPosition) > 3) {
stop(); identity = nextIdentity;
try { context ||= new (window.AudioContext || window.webkitAudioContext)({sampleRate: 48000}); context.resume().catch(error); }
@@ -414,6 +442,6 @@ export function createAudioPreview({state, sources, api, volumePreview = new Map
}
previousPosition = position; return true;
},
- dispose() { disposed = true; stop(); cache.clear(); context?.close().catch(() => {}); },
+ dispose() { disposed = true; stop(); cache.clear(); speech?.dispose(); context?.close().catch(() => {}); },
};
}
diff --git a/web/editor_audio_model.js b/web/editor_audio_model.js
index 754cd1f..5d8a52e 100644
--- a/web/editor_audio_model.js
+++ b/web/editor_audio_model.js
@@ -16,6 +16,18 @@ export function audioClipFields(raw, kind) {
if (typeof raw.detached_from !== 'string' || !/^c[1-9]\d{0,8}$/.test(raw.detached_from) || raw.detached_from === raw.id || raw.link) fail('link');
fields.detached_from = raw.detached_from;
}
+ // A music piece's preset (what it was made with, prefilled when it is made again) and whether it is
+ // lowered while the first sound track is heard.
+ if (raw.music !== undefined) {
+ const value = raw.music;
+ if (kind !== 'audio' || !value || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).sort().join() !== 'mood,style'
+ || ![value.style, value.mood].every(name => typeof name === 'string' && /^[a-z]{1,20}$/.test(name))) fail('audio');
+ fields.music = {style: value.style, mood: value.mood};
+ }
+ if (raw.duck !== undefined) {
+ if (kind !== 'audio' || typeof raw.duck !== 'boolean') fail('audio');
+ fields.duck = raw.duck;
+ }
return fields;
}
export function audioTrackFields(raw) {
@@ -97,6 +109,19 @@ export function setFades(record, ids, name, frames) {
}
return model.parse(next);
}
+// Lower during speech is a property of generated music only.
+export const duckable = clip => !!clip?.music && !!clip.source;
+export function setDuck(record, ids, on) {
+ if (typeof on !== 'boolean' || !ids.length) fail('audio');
+ editable(record, ids);
+ const next = model.serialize(record);
+ for (const id of ids) {
+ const clip = model.find(next, id);
+ if (trackOf(next, id)?.kind !== 'audio' || !duckable(clip)) fail('audio');
+ clip.duck = on;
+ }
+ return model.parse(next);
+}
export function setCrossfade(record, id, frames) {
if (!whole(frames)) fail('audio');
const next = model.serialize(record), track = trackOf(next, id), right = model.find(next, id);
@@ -205,3 +230,49 @@ export function applyTrackAudio(block, blockStart, track, tracks, {rate = 48000,
output.forEach((channel, index) => { for (let i = 0; i < count; i++) channel[i] += added[index][i]; });
return output;
}
+// Lower during speech: a clip with duck true sits DUCK_DB lower wherever another audible clip has speech,
+// with DUCK_RAMP_SECONDS ramps just outside each stretch. freevideo_engine/editor_audio.py has the same rules.
+export const DUCK_DB = -8, DUCK_RAMP_SECONDS = .3;
+const changedSpeed = clip => (clip.speed ?? 1) !== 1 || !!clip.reverse;
+// spans: Map or object of 'kind:id' → [[start, end] seconds]; ranges are timeline samples [first, last).
+export function duckRanges(tracks, spans, rate = 48000, fps = 24) {
+ const found = [], lookup = key => (spans instanceof Map ? spans.get(key) : spans?.[key]) || [];
+ for (const track of tracks) {
+ if (track.kind !== 'audio' || !trackAudible(track, tracks)) continue;
+ for (const clip of track.clips) {
+ if (!clip.source || clip.duck || (clip.volume ?? 1) === 0 || !clipAudible(clip, track, tracks)) continue;
+ const first = samples(clip.start, rate, fps), last = samples(end(clip), rate, fps);
+ const origin = changedSpeed(clip) || clip.speed_phase ? (clip.in * 2000 + (clip.speed_phase || 0) * 20) * rate / 48000 : sourceStart(clip, rate, fps);
+ const speed = changedSpeed(clip) ? (clip.speed ?? 1) : 1;
+ for (const [begin, stop] of lookup(`${clip.source.kind}:${clip.source.id}`)) {
+ let a = (begin * rate - origin) / speed, b = (stop * rate - origin) / speed;
+ if (clip.reverse) [a, b] = [(last - first) - b, (last - first) - a];
+ a = Math.max(first, Math.floor(first + a + .5)); b = Math.min(last, Math.floor(first + b + .5));
+ if (b > a) found.push([a, b]);
+ }
+ }
+ }
+ found.sort((x, y) => x[0] - y[0] || x[1] - y[1]);
+ const gap = 2 * Math.max(1, Math.floor(DUCK_RAMP_SECONDS * rate + .5)), joined = [];
+ for (const [a, b] of found) {
+ if (joined.length && a <= joined.at(-1)[1] + gap) joined.at(-1)[1] = Math.max(joined.at(-1)[1], b);
+ else joined.push([a, b]);
+ }
+ return joined;
+}
+export function duckGain(ranges, blockStart, count, rate = 48000) {
+ const gain = new Float64Array(count).fill(1), ramp = Math.max(1, Math.floor(DUCK_RAMP_SECONDS * rate + .5)), low = 10 ** (DUCK_DB / 20);
+ for (const [first, last] of ranges) {
+ if (last + ramp <= blockStart || first - ramp >= blockStart + count) continue;
+ for (let i = 0; i < count; i++) {
+ const position = blockStart + i;
+ const into = Math.max(0, Math.min(1, (position - (first - ramp)) / ramp)), outOf = Math.max(0, Math.min(1, ((last + ramp) - position) / ramp));
+ gain[i] = Math.min(gain[i], 1 - (1 - low) * Math.min(into, outOf));
+ }
+ }
+ return gain;
+}
+// The lowered clips' sources need speech spans from the clips they sit under.
+export const duckSources = tracks => tracks.some(track => track.clips.some(clip => clip.duck && clip.source))
+ ? [...new Set(tracks.filter(track => track.kind === 'audio').flatMap(track => track.clips)
+ .filter(clip => clip.source && !clip.duck).map(clip => `${clip.source.kind}:${clip.source.id}`))] : [];
diff --git a/web/editor_genbar.js b/web/editor_genbar.js
index b6992f0..02a35e4 100644
--- a/web/editor_genbar.js
+++ b/web/editor_genbar.js
@@ -6,6 +6,7 @@ import {createRequestAssets} from './request_assets.js';
import {issueButton} from './issue_button.js';
import {waitingText} from './chain_support.js';
import {showEmptyPrompt} from './empty_prompt_notice.js';
+import {segmented} from './editor_adjust.js';
const el = (tag, cls, text) => { const e = document.createElement(tag); if (cls) e.className = cls; if (text != null) e.textContent = text; return e; };
export const timeEstimate = (seconds, t) => seconds == null ? '' : seconds < 60
? t(`about ${Math.max(1, Math.ceil(seconds))} s`, `约 ${Math.max(1, Math.ceil(seconds))} 秒`)
@@ -65,6 +66,36 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {
if (assets) host.prepend(assets.element);
const firstFrame = createFirstFrameSwitch(host, {state, t, actions});
description.after(firstFrame.element);
+ // On the music track: a style and a mood above the description, each a row; the music's own first-use download
+ // in the same line above the bar as the video's.
+ const music = actions.music || null;
+ const presets = music ? el('div', 'fv-ed-music-presets') : null;
+ const musicAssets = music ? el('div', 'fv-request-assets fv-muted') : null;
+ let musicBytes = null, musicBytesAt = 0;
+ const size = bytes => bytes >= 1e9 ? (bytes / 1e9).toFixed(1) + '\u00a0GB' : Math.max(1, Math.round(bytes / 1e6)) + '\u00a0MB';
+ const pickers = {};
+ if (music) {
+ for (const [key, label, options] of [['style', t('Style', '风格'), music.styles()], ['mood', t('Mood', '情绪'), music.moods()]]) {
+ const row = el('div', 'fv-ed-music-row'), name = el('span', 'fv-ed-music-label', label);
+ const picker = segmented(options.map(([value, text]) => ({value, label: text})), {onChange: value => { music.choose({[key]: value}); render(); }});
+ picker.setAttribute('aria-label', label); pickers[key] = picker;
+ row.append(name, picker); presets.append(row);
+ }
+ presets.hidden = true; musicAssets.hidden = true;
+ description.before(presets); host.prepend(musicAssets);
+ }
+ async function showMusicAssets(shown) {
+ if (!music) return;
+ if (!shown) { musicAssets.hidden = true; return; }
+ // Asked again at most every 5 s while a stretch of the music track is chosen.
+ if (musicBytes === null || Date.now() - musicBytesAt > 5000) {
+ musicBytesAt = Date.now();
+ musicBytes = await music.downloadBytes();
+ if (disposed) return;
+ }
+ musicAssets.textContent = musicBytes > 0 ? t(`Downloads before generating: music model ${size(musicBytes)}`, `生成前会先下载:配乐模型 ${size(musicBytes)}`) : '';
+ musicAssets.hidden = !(musicBytes > 0) || host.dataset.mode !== 'selected' || !music.target();
+ }
const failure = el('div', 'fv-ed-gen-failure fv-result-links'), failureBar = el('div', 'fv-ed-failure-actions fv-actions');
const failureReason = el('span', 'fv-ed-failure-reason'); failureReason.setAttribute('role', 'status');
const report = el('button', 'fv-ed-failure-report fv-action fv-action-secondary', t('Download diagnostic report', '下载诊断报告')); report.type = 'button';
@@ -119,9 +150,11 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {
}
failure.hidden = !selectedFailure;
failureReason.textContent = selectedFailure?.text || '';
- report.hidden = !selectedFailure;
+ // A piece of music has no generation receipt to report.
+ report.hidden = !selectedFailure || selectedFailure.report === false;
const running = ['queued', 'running', 'cancelling'].includes(status?.state);
- const selected = !!selection && (!!previewSelection || (actions.supportedSelection?.() ?? true));
+ const musical = !!music?.target(selection);
+ const selected = !!selection && (!!previewSelection || musical || (actions.supportedSelection?.() ?? true));
// While a part generates, choosing another place offers Generate again; it is queued behind.
// A press is being sent for the parts in `pressing`; only that same selection waits for it.
const sending = pressing.has(JSON.stringify(state.selection ?? null)), starting = pressing.size > 0 && !running;
@@ -134,13 +167,13 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {
const stopped = ['failed', 'cancelled', 'interrupted'].includes(status?.state);
host.dataset.mode = busy ? 'running' : selected ? 'selected' : 'idle';
hint.hidden = !busy && !other;
- // The description and Generate stay where they are while a part is made, greyed, so nothing around them moves.
- input.hidden = generate.hidden = false;
+ // The description and Generate stay where they are while a part is made, greyed, so nothing around them moves;
+ // music being made shows only its progress and Cancel.
+ input.hidden = false; generate.hidden = musical && busy;
// A choice that will move to an empty spot can still be described first.
const writable = selected || !previewSelection && !!selection && !!actions.selectionMovable?.();
input.disabled = !writable || busy;
- input.placeholder = writable ? t('Describe this part', '描述这一段')
- : '';
+ input.placeholder = !writable ? '' : musical ? t('Describe the music', '描述配乐') : t('Describe this part', '描述这一段');
input.setAttribute('aria-label', t('Description', '描述'));
generate.classList.toggle('fv-primary', selected && !busy);
// A cancel already accepted is not offered again; the bar says it is stopping until the run ends.
@@ -170,14 +203,26 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {
hint.title = shown.map(([, text]) => text).join('');
}
firstFrame.update(busy);
+ if (presets) {
+ presets.hidden = !musical || !selected && !busy;
+ if (!presets.hidden) {
+ const value = music.preset(selection);
+ for (const key of ['style', 'mood']) { pickers[key].value = value[key]; pickers[key].disabled = busy || !!previewSelection; }
+ }
+ host.classList.toggle('fv-ed-genbar-music', !presets.hidden);
+ }
references?.update(busy, busy ? undefined : actions.references.visible() || false);
- generate.textContent = t('Generate', '生成');
+ // A piece already made, failed or stopped is made again in place.
+ const made = musical && !!music.target(selection)?.clip && music.target(selection).clip.pending !== 'draft';
+ const verb = made ? t('Regenerate', '重新生成') : t('Generate', '生成');
+ generate.textContent = verb;
// Never disabled for a reason the press itself can explain or fix; only until the server says it can generate.
generate.disabled = busy || !!previewSelection || !!actions.generationWaiting?.();
resizeDescription();
input.dispatchEvent(new Event('change'));
generate.title = '';
- if (assets) { if (selected && !busy) assets.refresh(); else assets.element.hidden = true; }
+ if (assets) { if (selected && !busy && !musical) assets.refresh(); else assets.element.hidden = true; }
+ showMusicAssets(selected && !busy && musical && !previewSelection);
if (selected && !busy) {
const key = JSON.stringify(selection);
if (estimateCache?.key === key) generate.textContent += estimateCache.label;
@@ -185,7 +230,7 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {
if (disposed || current !== serial) return;
const label = seconds == null ? '' : ' · ' + timeEstimate(seconds, t);
estimateCache = {key, label};
- generate.textContent = t('Generate', '生成') + label;
+ generate.textContent = verb + label;
}
}
function resizeDescription() {
@@ -281,7 +326,7 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {
if (!value && note.contains(retryService)) { retryService.remove(); note.hidden = true; }
showService();
}, getText() { expand.commit(); return input.value; }, update(value) { status = value; render(); }, preview(selection) { previewSelection = selection; render(); }, setText(value) { input.value = value || ''; state.setRangeText(input.value); if (bound && input.value !== bound.text) { dirty = true; written = input.value; } render(); },
- dispose() { disposed = true; ++serial; clearTimeout(noteTimer); clearTimeout(refNoticeTimer); unsubscribe(); resize?.disconnect(); expand.dispose(); references?.dispose(); assets?.dispose(); }};
+ dispose() { disposed = true; ++serial; clearTimeout(noteTimer); clearTimeout(refNoticeTimer); unsubscribe(); resize?.disconnect(); expand.dispose(); references?.dispose(); assets?.dispose(); Object.values(pickers).forEach(picker => picker.dispose()); }};
}
// A missing optional module leaves the editor's existing progress in place.
diff --git a/web/editor_lifecycle.js b/web/editor_lifecycle.js
index eed7b51..4054e09 100644
--- a/web/editor_lifecycle.js
+++ b/web/editor_lifecycle.js
@@ -6,9 +6,11 @@ export function discardEmptyEdit(record, initial, {busy = false, hasMedia = fals
if (busy || hasMedia || record.keep_empty || !untitled(record.name)
|| record.tracks.some(track => track.clips.length) || record.replacements?.length) return false;
// Compare settings, track controls and sparse project fields with initial().
+ // An untouched empty second sound track does not make an edit differ from a new one.
+ const bare = track => track.id === 'a2' && !track.clips.length && Object.keys(track).every(key => ['id', 'kind', 'clips'].includes(key));
const content = value => {
const {id, revision, serial, name, ...rest} = value;
- return rest;
+ return {...rest, tracks: rest.tracks.filter(track => !bare(track))};
};
return JSON.stringify(sorted(content(record))) === JSON.stringify(sorted(content(initial)));
}
diff --git a/web/editor_music.js b/web/editor_music.js
new file mode 100644
index 0000000..ff39c66
--- /dev/null
+++ b/web/editor_music.js
@@ -0,0 +1,177 @@
+// Music on the second sound track through the generation bar: a stretch chosen there (or a piece of music
+// there) is what Generate makes, with the bar's own description box, a style and a mood. The pieces follow the
+// music job's progress and take its file in place; the bar shows the job of the piece chosen.
+import * as music from './editor_music_model.js';
+import {EditError} from './editor_model.js';
+import {timeEstimate} from './editor_genbar.js';
+
+export const STYLE_NAMES = {cinematic: ['Cinematic', '电影感'], pop: ['Pop', '流行'], electronic: ['Electronic', '电子'],
+ piano: ['Piano', '钢琴'], folk: ['Folk', '民谣'], ambient: ['Ambient', '氛围']};
+export const MOOD_NAMES = {calm: ['Calm', '舒缓'], upbeat: ['Upbeat', '明快'], tense: ['Tense', '紧张'], warm: ['Warm', '温暖'], sad: ['Sad', '悲伤']};
+
+// What the bar says about a piece's job: queued, downloading on first use, waiting for a video, or composing.
+export function musicBarStatus(job, t, clock = Date.now) {
+ if (!job || !['queued', 'running'].includes(job.state)) return null;
+ let stage = '', time = '';
+ if (job.state === 'queued') stage = t('Queued', '排队中');
+ else if (job.phase === 'downloading') stage = t('Downloading files for generation', '正在下载生成所需文件');
+ else if (job.phase === 'waiting') stage = t('Queued · waiting for the previous one', '排队中 · 等前一个完成');
+ else if (job.phase === 'composing') {
+ stage = t('Generating music', '正在生成配乐');
+ // The server's elapsed time, carried forward between polls.
+ const elapsed = Number.isFinite(job.elapsed) ? job.elapsed + Math.max(0, (clock() - (job.seen_at ?? clock())) / 1000) : null;
+ if (Number.isFinite(job.estimate) && elapsed != null) time = elapsed > job.estimate * 1.2 + 5
+ ? t('Taking longer than expected', '比预计久,仍在生成') : timeEstimate(Math.max(1, job.estimate - elapsed), t);
+ } else stage = t('Preparing', '正在准备');
+ return {state: 'running', blocks: [], music: true, words: {stage, ...(time ? {time} : {})}};
+}
+
+// A failed piece's sentence on the bar's failure line.
+export function musicFailureText(reason, t) {
+ return {
+ music_download_failed: t('Download paused. Retry to resume.', '下载已暂停,重试可继续。'),
+ disk_space: t('The disk is full. Free some space, then generate again.', '磁盘空间不足,请清理出空间后再生成。'),
+ out_of_memory: t('Not enough memory to generate the music. Close other programs that use the graphics card, then generate again.', '生成配乐时内存不足,请关闭其他占用显卡的程序后重新生成。'),
+ unsupported: t('Music cannot be generated on this computer yet.', '这台电脑暂不能生成配乐。'),
+ missing: t('Generation stopped. Generate again.', '生成已中断,请重新生成。'),
+ }[reason] || t('The music could not be generated. Generate again.', '无法生成配乐,请重新生成。');
+}
+
+export function createEditorMusic({state, store, t, completeSources = async () => {}, onStatus = () => {}, clock = Date.now, random = Math.random}) {
+ const statuses = new Map(), watching = new Map();
+ let info = null, infoAsked = null, chosen = null, chosenFor = null, disposed = false;
+ const target = (selection = state.selection) => music.musicTarget(state.record, selection);
+ const refreshInfo = () => infoAsked ??= Promise.resolve().then(() => store.musicInfo()).then(value => { info = value || {available: false}; return info; })
+ .catch(() => (info = null)).finally(() => { infoAsked = null; onStatus(); });
+ // The style and mood the bar shows: the chosen piece's own, else the edit's last choice; a pick on the bar holds
+ // until the choice moves elsewhere.
+ function preset(selection = state.selection) {
+ const key = JSON.stringify(selection ?? null);
+ if (chosen && chosenFor === key) return chosen;
+ const found = target(selection);
+ return found?.clip?.music && music.MUSIC_STYLES.includes(found.clip.music.style) && music.MUSIC_MOODS.includes(found.clip.music.mood)
+ ? {...found.clip.music} : music.lastPreset(state.record);
+ }
+ function choose(value) {
+ chosen = {...preset(), ...value}; chosenFor = JSON.stringify(state.selection ?? null);
+ }
+ // A piece removed or changed meanwhile is left as it is.
+ const settle = change => { try { state.settle(change); } catch (value) { if (!(value instanceof EditError)) throw value; } };
+ function deliver(id, jobId, job) {
+ if (disposed || !job) return;
+ statuses.set(jobId, {...job, seen_at: clock()});
+ if (job.state === 'done' && job.result) {
+ stopWatching(jobId);
+ settle(record => music.finishMusic(record, id, jobId, {source: job.result.source, frames: job.result.frames}));
+ completeSources().catch(() => {});
+ // The piece just made sets this machine's pace, so the next Generate can show its estimate.
+ refreshInfo();
+ } else if (['failed', 'cancelled'].includes(job.state)) {
+ stopWatching(jobId);
+ settle(record => music.failMusic(record, id, jobId, job.state === 'cancelled' ? 'cancelled' : job.reason || 'music_failed'));
+ } else settle(record => music.musicProgress(record, id, jobId, job.state));
+ onStatus();
+ }
+ const starting = new Set(); let pressing = false;
+ function stopWatching(jobId) { watching.get(jobId)?.(); watching.delete(jobId); }
+ function watch(id, jobId) {
+ if (watching.has(jobId)) return;
+ watching.set(jobId, store.watchMusic(jobId, job => deliver(id, jobId, job), () => deliver(id, jobId, {id: jobId, state: 'failed', reason: 'missing'})));
+ }
+ async function begin(id, words) {
+ state.select({kind: 'clip', ids: [id]});
+ chosen = null;
+ let job;
+ try { job = await store.startMusic(music.musicRequest(state.record, id, {seed: Math.floor(random() * 2 ** 32)})); }
+ catch (value) {
+ settle(record => music.failMusic(record, id, undefined, value?.reason === 'unsupported' ? 'unsupported' : 'music_failed'));
+ onStatus();
+ return {say: musicFailureText(value?.reason === 'unsupported' ? 'unsupported' : 'music_failed', t)};
+ }
+ if (!job?.id) { settle(record => music.failMusic(record, id, undefined)); onStatus(); return {say: musicFailureText('music_failed', t)}; }
+ settle(record => music.assignMusicJob(record, id, job.id));
+ statuses.set(job.id, {...job, seen_at: clock()});
+ watch(id, job.id); onStatus();
+ return {text: words};
+ }
+ const api = {
+ target, preset, choose, refreshInfo,
+ get info() { return info; },
+ styles: () => music.MUSIC_STYLES.map(id => [id, t(...STYLE_NAMES[id])]),
+ moods: () => music.MUSIC_MOODS.map(id => [id, t(...MOOD_NAMES[id])]),
+ // The chosen piece is being made.
+ running(selection = state.selection) {
+ const found = target(selection);
+ return !!found?.clip && (starting.has(found.clip.id) || ['queued', 'running'].includes(found.clip.pending));
+ },
+ // The bar's status for the chosen piece's job; null when it has none running.
+ barStatus(selection = state.selection) {
+ const found = target(selection), job = found?.clip?.job && statuses.get(found.clip.job);
+ return job && ['queued', 'running'].includes(found.clip.pending) ? musicBarStatus(job, t, clock) : null;
+ },
+ // The words the box shows for a piece of music; null for a stretch not made yet.
+ text(selection = state.selection) {
+ const found = target(selection);
+ return found?.clip && !api.running(selection) ? {ids: [found.clip.id], text: found.clip.text} : null;
+ },
+ describe(id, text) {
+ const found = target({kind: 'clip', ids: [id]});
+ if (!found?.clip || api.running({kind: 'clip', ids: [id]}) || found.clip.text === text) return false;
+ state.apply(t('Edit description', '修改说明'), record => {
+ const next = JSON.parse(JSON.stringify(record));
+ next.tracks.find(track => track.id === music.MUSIC_TRACK).clips.find(clip => clip.id === id).text = text;
+ return next;
+ });
+ return true;
+ },
+ failure(selection = state.selection) {
+ const found = target(selection);
+ return found?.clip?.pending === 'failed' ? {id: found.clip.id, text: musicFailureText(found.clip.reason, t), report: false} : null;
+ },
+ // Seconds the piece will take: this machine's pace from its last piece; unknown before the first.
+ estimate(selection = state.selection) {
+ const found = target(selection);
+ return found && Number.isFinite(info?.pace) ? info.pace * found.frames / 24 : null;
+ },
+ // First-use download for the bar's "Downloads before generating" line: bytes still to fetch, or 0.
+ async downloadBytes() {
+ const pack = await Promise.resolve().then(() => store.musicPack()).catch(() => null);
+ return pack && !pack.ready && !pack.running ? pack.bytes || 0 : 0;
+ },
+ // Generate on a stretch of the music track, or again on a piece there. {say} answers the press.
+ // A second press while the first is still checking or starting its job is the same press.
+ async press(text) {
+ const found = target();
+ if (!found) return null;
+ if (pressing || api.running()) return {say: t('This part is generating.', '这一段正在生成。')};
+ pressing = true;
+ let id = found.id, words;
+ try {
+ const current = info || await refreshInfo();
+ if (current && current.available === false) return {say: musicFailureText('unsupported', t)};
+ const pack = await Promise.resolve().then(() => store.musicPack()).catch(() => null);
+ if (pack && !pack.ready && pack.disk_ok === false) return {say: musicFailureText('disk_space', t)};
+ const pick = preset(); words = text.trim();
+ state.apply(t('Generate', '生成'), record => {
+ const next = music.rememberPreset(record, pick);
+ if (id) return music.remakeMusic(next, id, {text: words, ...pick});
+ const planned = music.planMusic(next, {start: found.start, frames: found.frames, text: words, ...pick});
+ id = planned.id; return planned.record;
+ }, {selectionAfter: null});
+ starting.add(id);
+ } finally { pressing = false; }
+ try { return await begin(id, words); } finally { starting.delete(id); }
+ },
+ async cancel(selection = state.selection) {
+ const found = target(selection);
+ if (!found?.clip?.job || !api.running(selection)) return false;
+ deliver(found.clip.id, found.clip.job, await store.cancelMusic(found.clip.job).then(value => value?.job || value));
+ return true;
+ },
+ // Pieces still waiting for their job when the edit opens again.
+ recover() { for (const {id, job} of music.musicJobs(state.record)) watch(id, job); },
+ reset() { for (const jobId of [...watching.keys()]) stopWatching(jobId); statuses.clear(); chosen = null; },
+ dispose() { disposed = true; api.reset(); },
+ };
+ return api;
+}
diff --git a/web/editor_music_model.js b/web/editor_music_model.js
new file mode 100644
index 0000000..2f020e9
--- /dev/null
+++ b/web/editor_music_model.js
@@ -0,0 +1,132 @@
+// A piece of music planned on a stretch of the second sound track, made by the music job and laid back in place.
+// Pure edits on the project record; the generation bar decides when to call them.
+import * as model from './editor_model.js';
+import {FPS} from './editor_model.js';
+
+export const MUSIC_TRACK = 'a2';
+// The longest piece the music model makes in one go.
+export const MAX_MUSIC_FRAMES = 600 * FPS;
+// The longest description the music job takes.
+export const MAX_MUSIC_TEXT = 512;
+// The presets the server knows; the user's own words go instead when given.
+export const MUSIC_STYLES = ['cinematic', 'pop', 'electronic', 'piano', 'folk', 'ambient'];
+export const MUSIC_MOODS = ['calm', 'upbeat', 'tense', 'warm', 'sad'];
+const fail = code => { throw new model.EditError(code); };
+const end = clip => clip.start + clip.frames;
+const lane = record => record.tracks.find(track => track.id === MUSIC_TRACK) || null;
+
+// The music track's planned or running pieces: no file yet, waiting for (or made by) a music job.
+export const isMusicPart = (record, clip) => !!clip && !clip.source && clip.pending !== null
+ && !!lane(record)?.clips.some(other => other.id === clip.id);
+// A finished piece the music job made: its file sits in the music folder.
+export const isGeneratedMusic = clip => clip?.source?.kind === 'generated' && /^freevideo-music\/[a-z]+(-[a-z]+)?-[0-9a-f]{32}\.mp4$/.test(clip.source.id);
+
+export const DEFAULT_PRESET = {style: 'cinematic', mood: 'calm'};
+const preset = value => MUSIC_STYLES.includes(value?.style) && MUSIC_MOODS.includes(value?.mood) ? {style: value.style, mood: value.mood} : null;
+// The style and mood last chosen in this edit; the default pair before any.
+export const lastPreset = record => preset(record.settings?.music) || {...DEFAULT_PRESET};
+// Remember the pair for the next piece in this edit.
+export function rememberPreset(record, value) {
+ const chosen = preset(value) || fail('preset');
+ if (JSON.stringify(record.settings?.music) === JSON.stringify(chosen)) return model.parse(record);
+ const next = model.serialize(record);
+ next.settings = {...(next.settings || {}), music: chosen};
+ return model.parse(next);
+}
+// What a selection on the music track asks for: an empty stretch drawn or clicked there, or one piece of music
+// there (made again in place). Null for anything else.
+export function musicTarget(record, selection) {
+ const track = lane(record);
+ if (!track || !selection) return null;
+ if (selection.kind === 'range' && selection.tracks?.length === 1 && selection.tracks[0] === MUSIC_TRACK
+ || selection.kind === 'gap' && selection.track === MUSIC_TRACK) {
+ const frames = Math.min(selection.frames, MAX_MUSIC_FRAMES);
+ if (track.clips.some(clip => clip.start < selection.start + frames && end(clip) > selection.start)) return null;
+ return {start: selection.start, frames};
+ }
+ if (selection.kind !== 'clip' || selection.ids.length !== 1) return null;
+ const clip = track.clips.find(clip => clip.id === selection.ids[0]);
+ if (!clip || !(isMusicPart(record, clip) || isGeneratedMusic(clip))) return null;
+ return {start: clip.start, frames: clip.frames, id: clip.id, clip};
+}
+
+// A drag on the music track: the stretch from start for frames, which must be empty there. The track is added when missing.
+export function planMusic(record, {start, frames, text = '', style, mood}) {
+ if (!Number.isSafeInteger(start) || start < 0 || !Number.isSafeInteger(frames) || frames < 1) fail('range');
+ if (frames > MAX_MUSIC_FRAMES) fail('too_long');
+ if (typeof text !== 'string') fail('text');
+ const chosen = style === undefined && mood === undefined ? {...DEFAULT_PRESET} : preset({style, mood}) || fail('preset');
+ const next = model.serialize(lane(record) ? record : model.addAudioTrack(record)), track = next.tracks.find(track => track.id === MUSIC_TRACK);
+ if (track.locked) fail('locked');
+ if (track.clips.some(clip => clip.start < start + frames && end(clip) > start)) fail('overlap');
+ const id = `c${++next.serial}`;
+ track.clips.push({id, start, frames, in: 0, length: frames, source: null, link: null, text, part: null, pending: 'draft',
+ reason: null, job: null, extend: false, versions: [], volume: 1, music: chosen, duck: true});
+ track.clips.sort((a, b) => a.start - b.start);
+ return {record: model.parse(next), id};
+}
+// Made again in place: a finished, failed or stopped piece becomes a planned one over the same stretch, with the
+// words and preset given; its volume, fades and lowering stay. One undo brings the previous piece back.
+export function remakeMusic(record, id, {text = '', style, mood}) {
+ if (typeof text !== 'string') fail('text');
+ const chosen = preset({style, mood}) || fail('preset');
+ const next = model.serialize(record), clip = next.tracks.find(track => track.id === MUSIC_TRACK)?.clips.find(clip => clip.id === id);
+ if (!clip || !(isGeneratedMusic(clip) || !clip.source && clip.pending !== null)) fail('missing');
+ if (next.tracks.find(track => track.id === MUSIC_TRACK).locked) fail('locked');
+ if (['queued', 'running'].includes(clip.pending)) fail('busy');
+ Object.assign(clip, {source: null, in: 0, length: clip.frames, text, pending: 'draft', reason: null, job: null, versions: [], music: chosen});
+ delete clip.splice;
+ return model.parse(next);
+}
+
+// What the music job is asked for: the piece's length, its words when there are any, and its preset.
+export function musicRequest(record, id, {seed} = {}) {
+ const clip = model.find(record, id);
+ if (!isMusicPart(record, clip)) fail('missing');
+ const body = {frames: clip.frames, ...(clip.music || DEFAULT_PRESET)};
+ if (clip.text.trim()) body.text = clip.text.trim();
+ if (Number.isSafeInteger(seed)) body.seed = seed;
+ return body;
+}
+
+function change(record, id, job, edit) {
+ const next = model.serialize(record), clip = next.tracks.find(track => track.id === MUSIC_TRACK)?.clips.find(clip => clip.id === id);
+ if (!clip || clip.source || clip.pending === null) return null;
+ if (job !== undefined && clip.job !== job) return null;
+ edit(clip);
+ return model.parse(next);
+}
+// The job was accepted: the piece waits for it.
+export function assignMusicJob(record, id, job) {
+ if (typeof job !== 'string' || !/^m-[0-9a-f]{32}$/.test(job)) fail('job');
+ return change(record, id, undefined, clip => { clip.job = job; clip.pending = 'queued'; clip.reason = null; }) ?? fail('missing');
+}
+// What the job says while it runs; a piece already finished, removed or given to another job stays as it is.
+export function musicProgress(record, id, job, state) {
+ if (!['queued', 'running'].includes(state)) return model.parse(record);
+ return change(record, id, job, clip => { clip.pending = state; }) ?? model.parse(record);
+}
+// The file is made: it goes into the planned stretch. A shorter file shortens the piece; nothing else moves.
+export function finishMusic(record, id, job, {source, frames}) {
+ const [kind, path] = typeof source === 'string' ? [source.slice(0, source.indexOf(':')), source.slice(source.indexOf(':') + 1)] : [];
+ if (kind !== 'generated' || !path || !Number.isSafeInteger(frames) || frames < 1) fail('source');
+ return change(record, id, job, clip => {
+ Object.assign(clip, {source: {kind, id: path}, in: 0, length: frames, frames: Math.min(clip.frames, frames), pending: null, reason: null, job: null});
+ }) ?? model.parse(record);
+}
+// Failed or stopped: the piece stays where it was drawn, with its reason, ready to try again.
+export function failMusic(record, id, job, reason = 'music_failed') {
+ const state = reason === 'cancelled' ? 'stopped' : 'failed';
+ const code = typeof reason === 'string' && /^[a-z_]{1,80}$/.test(reason) ? reason : 'music_failed';
+ return change(record, id, job, clip => { clip.pending = state; clip.reason = state === 'failed' ? code : null; }) ?? model.parse(record);
+}
+// Try again: back to planned, with the same stretch and description.
+export function retryMusic(record, id) {
+ return change(record, id, undefined, clip => {
+ if (['queued', 'running'].includes(clip.pending)) fail('busy');
+ clip.pending = 'draft'; clip.reason = null; clip.job = null;
+ }) ?? fail('missing');
+}
+// The pieces still waiting for a job after the page opens again, to watch or to mark failed.
+export const musicJobs = record => (lane(record)?.clips || []).filter(clip => !clip.source && clip.job && ['queued', 'running'].includes(clip.pending))
+ .map(clip => ({id: clip.id, job: clip.job}));
diff --git a/web/editor_player.js b/web/editor_player.js
index 0524d7c..3a7cb78 100644
--- a/web/editor_player.js
+++ b/web/editor_player.js
@@ -27,7 +27,7 @@ export function selectAudioChain(chains, track, clip, position, volumePreview =
&& !c.clips.some(id => volumePreview.has(id) && volumePreview.get(id) !== 1));
return chain ? {chain, seconds: (position - chain.start) / 24 + (chain.offset || 0)} : null;
}
-export function createEditorPlayer(host, transport, {state, sources, api, t, error = () => {}, readAudioSamples = null, preservePreview = () => false, prepareAudio = () => true}) {
+export function createEditorPlayer(host, transport, {state, sources, api, t, error = () => {}, readAudioSamples = null, preservePreview = () => false, prepareAudio = () => true, speech = null}) {
const frame = el('div', 'fv-ed-picture'); frame.setAttribute('aria-label', t('Video preview', '视频预览'));
const empty = el('div', 'fv-ed-empty-preview'), pictureIcon = document.createElementNS('http://www.w3.org/2000/svg', 'svg');
pictureIcon.setAttribute('viewBox', '0 0 24 24'); pictureIcon.setAttribute('aria-hidden', 'true');
@@ -54,7 +54,7 @@ export function createEditorPlayer(host, transport, {state, sources, api, t, err
const chainSeconds = (chain, position) => Math.max(0, (position - chain.start) / 24 + (chain.offset || 0));
const playback = createPlaybackClock();
const gains = new WeakMap(), volumePreview = new Map();
- const audioPreview = createAudioPreview({state, sources, api, volumePreview, readSamples: readAudioSamples, error});
+ const audioPreview = createAudioPreview({state, sources, api, volumePreview, readSamples: readAudioSamples, error, speech});
const clipVolume = clip => volumePreview.get(clip?.id) ?? clip?.volume ?? 1;
function volume(media, value) {
// HTMLMediaElement stops at 1. A GainNode implements the record's 0–2 range.
diff --git a/web/editor_store.js b/web/editor_store.js
index 9590f9b..c95a28a 100644
--- a/web/editor_store.js
+++ b/web/editor_store.js
@@ -514,6 +514,33 @@ export function createEditorStore(api, {t = en => en, notify = () => {}, onGener
installEars: () => json('/freevideo/editor/subtitle-pack', post({action: 'install', language: true})),
cancelEarsPack: () => json('/freevideo/editor/subtitle-pack', post({action: 'cancel'})),
watchEarsPack(fn) { return watch(() => json('/freevideo/editor/subtitle-pack'), fn); },
+ // Music for a stretch of the sound track. {available, ready}; null where the server is older.
+ musicInfo: () => json('/freevideo/editor/music'),
+ // body: {frames, text?} or {frames, style, mood}, optional seed. Refusals keep the server's reason.
+ async startMusic(body) {
+ const response = await request('/freevideo/editor/music', post(body));
+ const value = await response.json().catch(() => null);
+ if (!response.ok) throw Object.assign(errorFor(response), {reason: value?.reason || (response.status === 404 ? 'unsupported' : 'failed')});
+ return value?.job || null;
+ },
+ // A job the server no longer knows (it was restarted) reads as failed, so the planned piece does not wait forever.
+ watchMusic(id, fn, lost) {
+ return watch(async () => (await json('/freevideo/editor/music?' + new URLSearchParams({id}), undefined, {job: {id, state: 'failed', reason: 'missing'}}))?.job || null, fn, lost);
+ },
+ cancelMusic: id => json('/freevideo/editor/music/cancel', post({id})),
+ // The music model is downloaded on first use, with the download source chosen in Settings.
+ musicPack: () => json('/freevideo/editor/music-pack'),
+ // Speech spans of one source in seconds, for "Lower during speech": {spans}, {state: 'pending'} or {spans: null, reason}.
+ speech: key => json('/freevideo/editor/speech?' + new URLSearchParams({source: key})),
+ async installMusic() {
+ const response = await request('/freevideo/editor/music-pack', post({action: 'install'}));
+ const value = await response.json().catch(() => null);
+ // 507: not enough room on the disk; the state that comes back says how much is needed.
+ if (!response.ok && response.status !== 507) throw Object.assign(errorFor(response), {reason: value?.reason || 'failed'});
+ return value;
+ },
+ cancelMusicPack: () => json('/freevideo/editor/music-pack', post({action: 'cancel'})),
+ watchMusicPack(fn) { return watch(() => json('/freevideo/editor/music-pack'), fn); },
dispose() { disposed = true; activateProject(null).catch(() => {}); session.dispose(); for (const poller of pollers) poller.dispose(); pollers.clear(); watchers.clear();
for (const upload of [...uploads]) { upload.release(); upload.xhr.abort(); upload.reject(new Error('editor_closed')); } },
};
diff --git a/web/editor_tracks.js b/web/editor_tracks.js
index e02d216..af51c30 100644
--- a/web/editor_tracks.js
+++ b/web/editor_tracks.js
@@ -13,6 +13,9 @@ import {attachAudioTracks} from './editor_audio.js';
import {seconds as secondsText} from './editor_audio_notes.js';
import {renderSpeedBadge} from './editor_speed.js';
import {speedEdited} from './editor_speed_model.js';
+import {MUSIC_TRACK, MAX_MUSIC_FRAMES} from './editor_music_model.js';
+// A range drawn on the music track alone: music for that stretch.
+const musicRange = selection => selection?.kind === 'range' && selection.tracks?.length === 1 && selection.tracks[0] === MUSIC_TRACK;
export const SOURCE_TYPE = 'application/x-freevideo-source';
export const SNAP_DISTANCE = 8;
@@ -280,11 +283,13 @@ export function createTracks(host, {state, sources = {}, t = en => en, actions =
}
// A range cannot be drawn longer than one video may be; `fromEnd` keeps its end where it is.
function capRange(selection, fromEnd = false, continues = false) {
- const most = model.maxRangeFrames(state.record, {...actions.generationLimits?.(), continues});
+ const most = musicRange(selection) ? MAX_MUSIC_FRAMES : model.maxRangeFrames(state.record, {...actions.generationLimits?.(), continues});
if (selection.frames <= most) return selection;
return fromEnd ? {...selection, start: selection.start + selection.frames - most, frames: most} : {...selection, frames: most};
}
function rangeParts(selection, record = state.record) {
+ // A stretch of the music track is one piece, never split into video parts.
+ if (musicRange(selection)) return [selection.frames];
return model.rangeParts(record, {...selection, ...actions.generationLimits?.()});
}
function boundaryLine(frame, index) {
@@ -806,7 +811,8 @@ export function createTracks(host, {state, sources = {}, t = en => en, actions =
const aligned = snap(frame, points, state.zoom);
// Where it was pressed, drawn to a clip edge, marker or the playhead nearby.
gesture.from ??= snap(gesture.anchor, points, state.zoom).frame;
- const drawn = rangeParameters(gesture.from, aligned.frame);
+ // Drawn on the music track, the range is that track's alone: music for that stretch.
+ const drawn = rangeParameters(gesture.from, aligned.frame, gesture.track === MUSIC_TRACK ? [MUSIC_TRACK] : undefined);
// Drawn from a clip's end, the range starts exactly there.
if (gesture.flush != null && aligned.frame > gesture.flush) Object.assign(drawn, {start: gesture.flush,
frames: Math.max(2 * model.FPS, Math.round((aligned.frame - gesture.flush) / model.FPS) * model.FPS)});