Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.
Editor: music for a stretch of the second sound track from the generation bar, lowered during speech
Bowen Xuecf720c649d81
diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.mdindex 66b947e..5f7e5ba 100644--- a/THIRD_PARTY_NOTICES.md+++ b/THIRD_PARTY_NOTICES.md@@ -19,7 +19,7 @@ FreeVideo code uses [Apache-2.0](LICENSE). Its dependencies and models retain th| [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) 1.13.8 speech runtime, built by FreeVideo without text-to-speech, ALSA and PortAudio (modified build; `__init__.py` edited; source snapshot 11afbd00 kept by FreeVideo; optional download for automatic subtitles, [build note](freevideo_engine/licenses/sherpa-onnx-build-NOTE.txt)). Includes statically linked kaldi-native-fbank, kaldi-decoder, kaldifst, OpenFst, simple-sentencepiece (Apache-2.0), KissFFT (BSD-3-Clause), pybind11 (BSD-3-Clause), nlohmann/json (MIT), hclust-cpp and Darts-clone (BSD-2-Clause), ThreadPool (Zlib). The Windows build also contains msvcp140.dll and msvcp140_1.dll (Microsoft Visual C++ 2022 runtime 14.44.35211, © Microsoft Corporation), unmodified and redistributed as permitted by Microsoft for its redistributable files. | Apache-2.0 || [ONNX Runtime](https://github.com/microsoft/onnxruntime) 1.28.2 (CPU), © Microsoft Corporation; its third-party notices are shipped inside the sherpa-onnx wheel | MIT || [Eigen](https://gitlab.com/libeigen/eigen) 5.0.1, compiled into sherpa-onnx and ONNX Runtime; source: https://gitlab.com/libeigen/eigen/-/tree/5.0.1 | MPL-2.0 |-| [Silero VAD](https://github.com/snakers4/silero-vad) v5.0 voice-activity model, © 2020-present Silero Team (optional download; license file downloaded with the model) | [MIT](freevideo_engine/licenses/Silero-VAD-MIT.txt) |+| [Silero VAD](https://github.com/snakers4/silero-vad) v5.0 voice-activity model, © 2020-present Silero Team (optional download with subtitles, license file downloaded with the model; the onnx-community export is downloaded with the music model) | [MIT](freevideo_engine/licenses/Silero-VAD-MIT.txt) || [NVIDIA NeMo stt_en_conformer_ctc_small](https://huggingface.co/nvidia/stt_en_conformer_ctc_small), © NVIDIA, exported to ONNX and quantized to int8 by the sherpa-onnx project, changed from the original; not endorsed by NVIDIA (optional download for English subtitles; [notice](freevideo_engine/licenses/NeMo-stt_en_conformer_ctc_small-NOTICE.txt)) | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) || [OpenAI Whisper tiny](https://github.com/openai/whisper), © 2022 OpenAI, exported to ONNX and quantized to int8 by the sherpa-onnx project (optional download for spoken-language detection) | [MIT](freevideo_engine/licenses/Whisper-MIT.txt) || [SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) by FunASR / FunAudioLLM (Alibaba Group), converted to ONNX and quantized to int8 by the sherpa-onnx project, used unchanged under its original name (optional download for multilingual subtitles; [notice](freevideo_engine/licenses/SenseVoice-NOTICE.txt)) | [FunASR Model Open Source License Agreement v1.1](freevideo_engine/licenses/FunASR-MODEL_LICENSE-v1.1.txt) |diff --git a/__init__.py b/__init__.pyindex 886ebff..3349199 100644--- a/__init__.py+++ b/__init__.py@@ -45,6 +45,8 @@ async def comfy_entrypoint():capabilities_routes()from .freevideo_engine.subtitle_ears import register as subtitle_ears_routessubtitle_ears_routes()+ from .freevideo_engine.editor_music import register as editor_music_routes+ editor_music_routes()from .freevideo_engine.subtitle_files import register as subtitle_file_routessubtitle_file_routes()from .freevideo_engine.comfy_share import register as share_routesdiff --git a/freevideo_engine/editor_audio.py b/freevideo_engine/editor_audio.pyindex d65a2af..ab605d8 100644--- a/freevideo_engine/editor_audio.py+++ b/freevideo_engine/editor_audio.py@@ -18,6 +18,17 @@ def clip_fields(raw, kind):or value == raw['id'] or raw.get('link')):raise ValueError('Invalid detached audio')result['detached_from'] = value+ # A music piece's preset (prefilled when it is made again) and whether it is lowered under the first sound track.+ if 'music' in raw:+ value = raw['music']+ if (kind != 'audio' or not isinstance(value, dict) or set(value) != {'style', 'mood'}+ or not all(isinstance(value[k], str) and re.fullmatch(r'[a-z]{1,20}', value[k]) for k in ('style', 'mood'))):+ raise ValueError('Invalid music preset')+ result['music'] = {'style': value['style'], 'mood': value['mood']}+ if 'duck' in raw:+ if kind != 'audio' or type(raw['duck']) is not bool:+ raise ValueError('Invalid audio ducking')+ result['duck'] = raw['duck']return result@@ -157,3 +168,82 @@ def apply_track_audio(block, block_start, track, tracks, *, rate=48000, fps=24,added[:, a - block_start:b - block_start] += (data[:, offset + a - begin:offset + b - begin]* gain(np.arange(a, b))[None, :] * clip.get('volume', 1))return output + added+++# Lower during speech: a clip with duck=True sits DUCK_DB lower wherever another audible clip has speech,+# with DUCK_RAMP_SECONDS ramps just outside each stretch. web/editor_audio_model.js has the same rules.+DUCK_DB = -8.+DUCK_RAMP_SECONDS = .3+++def _speed(clip):+ return clip.get('speed', 1) if changed(clip) else 1+++def duck_ranges(tracks, spans, rate=48000, fps=24):+ """Timeline sample ranges [first, last) with speech in a clip that is not itself lowered.++ spans maps 'kind:id' source keys to [start, end] seconds of that source; a source without an entry+ has no known speech. Ranges closer than two ramps are joined so the level does not bounce between them.+ """+ found = []+ for track in tracks:+ if track.get('kind') != 'audio' or not track_audible(track, tracks):+ continue+ for clip in track['clips']:+ source = clip.get('source')+ if not source or clip.get('duck') or clip.get('volume', 1) == 0 or not clip_audible(clip, track, tracks):+ continue+ stretches = spans.get(source['kind'] + ':' + source['id']) or ()+ first, last = _samples(clip['start'], rate, fps), _samples(clip['start'] + clip['frames'], rate, fps)+ origin = (source_sample_start(clip) * rate / 48000 if audio_changed(clip)+ else _source_start(clip, rate, fps))+ speed = _speed(clip)+ for begin, end in stretches:+ a, b = (begin * rate - origin) / speed, (end * rate - origin) / speed+ if clip.get('reverse'):+ a, b = (last - first) - b, (last - first) - a+ a, b = max(first, math.floor(first + a + .5)), min(last, math.floor(first + b + .5))+ if b > a:+ found.append([a, b])+ found.sort()+ ramp = 2 * max(1, math.floor(DUCK_RAMP_SECONDS * rate + .5))+ joined = []+ for a, b in found:+ if joined and a <= joined[-1][1] + ramp:+ joined[-1][1] = max(joined[-1][1], b)+ else:+ joined.append([a, b])+ return joined+++def duck_gain(ranges, block_start, count, rate=48000):+ """Gain per sample for a lowered clip over [block_start, block_start + count)."""+ import numpy as np+ gain = np.ones(count, dtype=np.float64)+ ramp = max(1, math.floor(DUCK_RAMP_SECONDS * rate + .5))+ low = 10. ** (DUCK_DB / 20.)+ position = np.arange(block_start, block_start + count, dtype=np.float64)+ for first, last in ranges:+ if last + ramp <= block_start or first - ramp >= block_start + count:+ continue+ into = np.clip((position - (first - ramp)) / ramp, 0, 1)+ out_of = np.clip(((last + ramp) - position) / ramp, 0, 1)+ np.minimum(gain, 1 - (1 - low) * np.minimum(into, out_of), out=gain)+ return gain+++def duck_sources(tracks):+ """Source keys whose speech spans a lowered clip needs (none when nothing is lowered)."""+ if not any(clip.get('duck') and clip.get('source') for track in tracks for clip in track['clips']):+ return []+ keys = []+ for track in tracks:+ if track.get('kind') != 'audio':+ continue+ for clip in track['clips']:+ source = clip.get('source')+ key = source and source['kind'] + ':' + source['id']+ if key and not clip.get('duck') and key not in keys:+ keys.append(key)+ return keysdiff --git a/freevideo_engine/editor_export.py b/freevideo_engine/editor_export.pyindex 671ddf2..01a0659 100644--- a/freevideo_engine/editor_export.py+++ b/freevideo_engine/editor_export.py@@ -28,7 +28,8 @@ from .editor_limits import PROJECT_BYTESfrom .editor_text import is_text_track, has_text_data, validate_text_project, paint_project, check_export_text, MissingGlyphsfrom . import editor_export_optionsfrom .editor_speed import validate_clip, changed as speed_changed, audio_changed, independent_join, VideoReader, AudioReader-from .editor_audio import clip_fields, track_fields, clip_audible, track_audible, crossfade_reads, apply_track_audio+from .editor_audio import (clip_fields, track_fields, clip_audible, track_audible, crossfade_reads, apply_track_audio,+ duck_ranges, duck_gain, duck_sources)FPS = 24SAMPLE_RATE = 48000@@ -693,6 +694,10 @@ def _render_once(project, resolve, target, *, progress, cancelled, options, repaaudio_tracks = {track['id']: track for track in project['tracks'] if track['kind'] == 'audio'}envelopes = {key: _CrossfadeReads(track, resolved.__getitem__, cancelled)for key, track in audio_tracks.items()}+ # Lower during speech: the same cached spans the preview reads.+ keys = duck_sources(project['tracks'])+ ducking = duck_ranges(project['tracks'], _speech_spans(+ keys, lambda key: (resolved[key] if key in resolved else resolve(key)).get('audio'), cancelled)) if keys else []def joinable(track):# A continuation joins its sound only where nothing else shapes that seam:@@ -800,6 +805,8 @@ def _render_once(project, resolve, target, *, progress, cancelled, options, repaor not clip_audible(following, audio_tracks[track], project['tracks'])or following['start'] != clip['start'] + clip['frames'])):gain *= np.clip((clip['frames'] * SAMPLES_PER_FRAME - 1 - positions) / (FADE_SAMPLES - 1), 0, 1).astype(np.float32)+ if clip.get('duck') and ducking:+ gain *= duck_gain(ducking, number * SAMPLES_PER_FRAME, SAMPLES_PER_FRAME).astype(np.float32)track_block = samples * gain[None, :] * clip.get('volume', 1)extras = (envelopes[track].extras(number * SAMPLES_PER_FRAME, SAMPLES_PER_FRAME)if track_audible(audio_tracks[track], project['tracks']) else None)@@ -853,6 +860,15 @@ def _render_once(project, resolve, target, *, progress, cancelled, options, repatarget.unlink(missing_ok=True)+def _speech_spans(keys, audio, cancelled):+ from .editor_music import SpeechSpans+ from .speech import SpeechError+ try:+ return SpeechSpans(_output_root(), resolve=audio).spans(keys, cancelled)+ except SpeechError:+ raise ExportCancelled() from None++def _output_root():import folder_pathsreturn Path(folder_paths.get_output_directory()).resolve()diff --git a/freevideo_engine/editor_music.py b/freevideo_engine/editor_music.pynew file mode 100644index 0000000..d087024--- /dev/null+++ b/freevideo_engine/editor_music.py@@ -0,0 +1,600 @@+"""Music for the editing page: a stretch of the sound track made into one piece of music.++The page plans the piece where the user dragged (start and length on the sound track), sends the+length and what the music should be (its own words, or a style and a mood), and lays the finished+file into that stretch. This module only makes the file: first-use download of the music model,+one job at a time, progress, cancel. The music service yields to video: a video that starts while+the music is made stops it, and the job waits for the video and composes again.+"""+import logging+import re+import secrets+import threading+import time+from collections import deque+from pathlib import Path++FPS = 24+FOLDER = 'freevideo-music'+MAX_JOBS = 20+MAX_QUIET = 500+MAX_TEXT = 512+# A video that starts while the music is made stops it; the job waits and composes again (as the Agent's job does).+PREEMPTIONS = 8+POLL_SECONDS = .5+_JOB_ID = re.compile(r'm-[0-9a-f]{32}')+_FIELDS = {'frames', 'text', 'style', 'mood', 'seed', 'quiet'}+REASONS = ('music_download_failed', 'out_of_memory', 'music_failed', 'invalid_request', 'busy', 'unsupported', 'failed')++# Presets: the genre and instruments, then the feeling and pace. Instrumental only, as the Agent asks for.+STYLES = {+ 'cinematic': 'Cinematic orchestral score with strings, soft brass and piano',+ 'pop': 'Modern instrumental pop with clean electric guitar, bass, light drums and synth pads',+ 'electronic': 'Electronic track with warm synthesizers, a steady drum machine and a deep bass line',+ 'piano': 'Solo piano piece with a clear melody and gentle sustain',+ 'folk': 'Acoustic folk tune with fingerpicked guitar, light percussion and a soft violin',+ 'ambient': 'Ambient soundscape with slow evolving pads, soft textures and distant bells',+}+MOODS = {+ 'calm': 'calm and peaceful, slow tempo around 70 BPM',+ 'upbeat': 'bright and upbeat, lively tempo around 120 BPM',+ 'tense': 'tense and suspenseful, a low pulse around 90 BPM that slowly builds',+ 'warm': 'warm and tender, moderate tempo around 85 BPM',+ 'sad': 'sad and reflective, slow tempo around 65 BPM in a minor key',+}+# The feeling alone, without the pace: used after the user's own words, which set the pace themselves+# (a fixed "around 70 BPM" after "a festival parade" pulled the piece back to the preset).+FEELINGS = {+ 'calm': 'calm and peaceful',+ 'upbeat': 'bright and upbeat',+ 'tense': 'tense and suspenseful',+ 'warm': 'warm and tender',+ 'sad': 'sad and reflective, in a minor key',+}+DEFAULT_STYLE, DEFAULT_MOOD = 'cinematic', 'calm'+++def caption(style, mood, words=None):+ """The English description the music model reads for a preset pair, or for the user's words with the pair after them."""+ if style not in STYLES or mood not in MOODS:+ raise ValueError('Unknown style or mood')+ if words:+ return '%s. %s, %s. Instrumental only, no singing, no vocals.' % (words.rstrip('.'), STYLES[style], FEELINGS[mood])+ return '%s, %s. Instrumental only, no singing, no vocals.' % (STYLES[style], MOODS[mood])+++def _whole(value, low, high):+ return type(value) is int and low <= value <= high+++def max_frames():+ from .music import service+ return service.MAX_SECONDS * FPS+++def validate_request(body):+ """The page's request -> what the job runs. Unknown fields, unknown presets and bad ranges are refused."""+ if not isinstance(body, dict) or set(body) - _FIELDS:+ raise ValueError('Invalid request')+ frames = body.get('frames')+ if not _whole(frames, 1, max_frames()):+ raise ValueError('Invalid length')+ text = body.get('text', '')+ if not isinstance(text, str) or len(text) > MAX_TEXT or '\x00' in text:+ raise ValueError('Invalid description')+ text = ' '.join(text.split())+ style, mood = body.get('style'), body.get('mood')+ if (style is None) != (mood is None):+ raise ValueError('Style and mood go together')+ if style is not None and (style not in STYLES or mood not in MOODS):+ raise ValueError('Unknown style or mood')+ if not text and style is None:+ style, mood = DEFAULT_STYLE, DEFAULT_MOOD+ seed = body.get('seed')+ if seed is None:+ seed = secrets.randbelow(2**32)+ elif not _whole(seed, 0, 2**32 - 1):+ raise ValueError('Invalid seed')+ quiet = body.get('quiet', [])+ if not isinstance(quiet, list) or len(quiet) > MAX_QUIET:+ raise ValueError('Invalid quiet spans')+ spans = []+ for span in quiet:+ if (not isinstance(span, dict) or set(span) != {'start', 'frames'} or not _whole(span['start'], 0, frames - 1)+ or not _whole(span['frames'], 1, frames - span['start'])):+ raise ValueError('Invalid quiet span')+ spans.append((span['start'], span['frames']))+ # The user's own words go first, with the chosen style and feeling after them (not the preset's pace);+ # the full preset alone when there are none.+ if text:+ words, stem = text if style is None else caption(style, mood, text), 'text'+ else:+ words, stem = caption(style, mood), '%s-%s' % (style, mood)+ return dict(frames=frames, caption=words, stem=stem, style=style, mood=mood, seed=seed, quiet=spans)+++def _gpu_busy(server):+ """True while a video (queued or running), a picture or the prompt model has the GPU."""+ try:+ from . import resident_process+ if resident_process.OWNER.active:+ return True+ except Exception: # noqa: BLE001 - an optional check; the music service refuses again if the GPU is taken+ pass+ if server is None:+ return False+ try:+ front = getattr(server, '_freevideo_front', None)+ running, pending = front.current_queue() if front is not None else server.prompt_queue.get_current_queue()+ if running or pending:+ return True+ except Exception: # noqa: BLE001+ pass+ for name in ('_freevideo_prompt_vlm',):+ service = getattr(server, name, None)+ try:+ if service is not None and service.busy():+ return True+ except Exception: # noqa: BLE001+ pass+ try:+ from .image_gen.service import get_service+ service = get_service(server)+ return service is not None and service.busy()+ except Exception: # noqa: BLE001+ return False+++def run_job(request, *, root, out, phase, cancelled, server=None, service=None, busy=None, poll=POLL_SECONDS):+ """Compose one piece into `out`; returns the music service's numbers. Raises InterruptedError when cancelled."""+ if service is None:+ from .music import service+ busy = busy or (lambda: _gpu_busy(server))+ for attempt in range(PREEMPTIONS + 1):+ try:+ return service.compose(out, request['caption'], request['frames'], seed=request['seed'], quiet=request['quiet'],+ fps=FPS, server=server, cancelled=cancelled, phase=phase)+ except service.Preempted:+ if attempt == PREEMPTIONS:+ raise service.MusicError('music_failed', 'preempted') from None+ phase('waiting')+ # Wait out the video (and anything queued with it) before the GPU is taken again.+ while True:+ if cancelled():+ raise InterruptedError('cancelled') from None+ time.sleep(poll)+ if not busy():+ break+++class MusicQueue:+ """One worker, first in first out, cancellable; keeps the last MAX_JOBS jobs for the page to read back."""+ def __init__(self, root, *, runner=None, read=None, clock=time.monotonic, **kwargs):+ self.root = Path(root).absolute()+ self.runner = runner or run_job+ self.read = read or self._read+ self.clock = clock+ self.kwargs = kwargs+ self.lock = threading.RLock()+ self.jobs = {}+ self.pending = deque()+ self.worker = None+ # Seconds of composing per second of music on this machine, from the last finished job.+ self.pace = None++ def _read(self, source):+ from . import comfy_editor+ return comfy_editor.sources(self.root, [source])[source]++ def _view(self, job):+ value = {key: job[key] for key in ('id', 'state', 'phase', 'fraction', 'reason', 'result')}+ # Composing reports no steps; the page shows time instead, from how long it took here last time.+ if job['state'] == 'running' and job['phase'] == 'composing' and job['began'] is not None:+ value['elapsed'] = round(self.clock() - job['began'], 1)+ value['estimate'] = None if self.pace is None else round(self.pace * job['request']['frames'] / FPS, 1)+ return value++ def submit(self, body):+ request = validate_request(body)+ hexid = secrets.token_hex(16)+ job = dict(id='m-' + hexid, state='queued', phase=None, fraction=0., reason=None, result=None, began=None,+ request=request, relative='%s/%s-%s.mp4' % (FOLDER, request['stem'], hexid), stop=threading.Event())+ with self.lock:+ for old in list(self.jobs):+ if len(self.jobs) < MAX_JOBS:+ break+ if self.jobs[old]['state'] not in ('queued', 'running'):+ del self.jobs[old]+ if len(self.jobs) >= MAX_JOBS:+ raise MusicBusy()+ self.jobs[job['id']] = job+ self.pending.append(job['id'])+ initial = self._view(job)+ if self.worker is None:+ self.worker = threading.Thread(target=self._run, name='freevideo-editor-music', daemon=True)+ self.worker.start()+ return initial++ def get(self, identity):+ if not isinstance(identity, str) or not _JOB_ID.fullmatch(identity):+ raise ValueError('Invalid job')+ with self.lock:+ if identity not in self.jobs:+ raise KeyError(identity)+ return self._view(self.jobs[identity])++ def cancel(self, identity):+ with self.lock:+ self.get(identity)+ job = self.jobs[identity]+ if job['state'] in ('queued', 'running'):+ job['stop'].set()+ if identity in self.pending:+ self.pending.remove(identity)+ job.update(state='cancelled', phase=None)+ return self._view(job)++ def busy(self):+ with self.lock:+ return any(job['state'] in ('queued', 'running') for job in self.jobs.values())++ def _run(self):+ while True:+ with self.lock:+ if not self.pending:+ self.worker = None+ return+ job = self.jobs[self.pending.popleft()]+ if job['stop'].is_set():+ job['state'] = 'cancelled'+ continue+ job['state'] = 'running'++ def phase(name, fraction=None):+ with self.lock:+ if name == 'composing' and job['phase'] != 'composing':+ job['began'] = self.clock()+ job['phase'] = name+ job['fraction'] = 0. if fraction is None else max(0., min(.99, float(fraction)))++ out = self.root / job['relative']+ try:+ out.parent.mkdir(parents=True, exist_ok=True)+ began = self.clock()+ numbers = self.runner(job['request'], root=self.root, out=out, phase=phase, cancelled=job['stop'].is_set,+ **self.kwargs) or {}+ if job['stop'].is_set():+ raise InterruptedError('cancelled')+ source = 'generated:' + job['relative']+ found = self.read(source)+ if found.get('missing') or not found.get('frames') or not found.get('has_audio'):+ raise MusicFailed('music_failed')+ spent = numbers.get('load_s', 0) + numbers.get('generate_s', 0) if numbers.get('generate_s') else self.clock() - began+ with self.lock:+ self.pace = max(.05, spent / (job['request']['frames'] / FPS))+ job.update(state='done', phase=None, fraction=1., result=dict(+ source=source, frames=int(found['frames']), seed=job['request']['seed'],+ style=job['request']['style'], mood=job['request']['mood']))+ except BaseException as error: # noqa: BLE001 - every outcome is the job's; the worker keeps serving+ out.unlink(missing_ok=True)+ with self.lock:+ if isinstance(error, InterruptedError) or job['stop'].is_set():+ job.update(state='cancelled', phase=None, reason=None)+ else:+ code = getattr(error, 'code', None)+ if code not in REASONS:+ logging.warning('[FreeVideo] Editor music failed', exc_info=error)+ code = 'music_failed'+ job.update(state='failed', phase=None, reason=code)+ if not isinstance(error, Exception):+ raise+++class MusicBusy(Exception):+ code = 'busy'+++class MusicFailed(Exception):+ def __init__(self, code):+ super().__init__(code)+ self.code = code+++class MusicPack:+ """First-use download of the music model, with the download sources chosen in Settings."""+ def __init__(self, *, installation=None, service=None, status=None, free_bytes=None):+ self._installation = installation+ self._service = service+ self._status = status+ self._free = free_bytes+ self.lock = threading.Lock()+ self.value = dict(running=False, fraction=0., reason=None)+ self.stop = threading.Event()+ self.worker = None++ def installation(self):+ if self._installation is not None:+ return self._installation()+ from .comfy_bridge import installation+ return installation()++ def service(self):+ if self._service is not None:+ return self._service+ from .music import service+ return service++ def status(self):+ root, machine = self.installation()+ if self._status is not None:+ pack = self._status(machine['model_root'])+ else:+ from .image_gen import image_pack+ pack = image_pack.status(machine['model_root'], group='music')+ if self._free is not None:+ free = self._free(machine['model_root'])+ else:+ from . import disk_space+ from .image_gen import image_pack+ folder = image_pack.base(machine['model_root'])+ while not folder.exists() and folder.parent != folder:+ folder = folder.parent+ free = disk_space.free_bytes(folder)+ remaining = int(pack.get('remaining_bytes', pack.get('bytes', 0)))+ from .image_gen.image_pack import RESERVE+ with self.lock:+ value = dict(self.value)+ value.update(ready=bool(pack['ready']), bytes=remaining,+ disk_ok=bool(pack['ready']) or free >= remaining + RESERVE)+ return value++ def start(self):+ """Begin the download unless it runs already; returns False when the disk cannot take it."""+ value = self.status()+ if value['ready'] or value['running']:+ return True+ if not value['disk_ok']:+ with self.lock:+ self.value.update(reason='disk_space')+ return False+ root, machine = self.installation()+ with self.lock:+ if self.value['running']:+ return True+ self.stop = threading.Event()+ self.value = dict(running=True, fraction=0., reason=None)+ self.worker = threading.Thread(target=self._run, args=(root, machine['model_root'], self.stop),+ name='freevideo-editor-music-pack', daemon=True)+ self.worker.start()+ return True++ def cancel(self):+ self.stop.set()++ def _run(self, root, model_root, stop):+ service = self.service()++ def progress(fraction):+ with self.lock:+ self.value['fraction'] = max(self.value['fraction'], min(.99, float(fraction)))++ reason = None+ try:+ service.install(model_root, root, progress, stop.is_set)+ except InterruptedError:+ pass+ except service.MusicError as error:+ reason = error.reason or 'failed'+ except Exception: # noqa: BLE001 - the page says it did not download; the log keeps why+ logging.warning('[FreeVideo] Music model download failed', exc_info=True)+ reason = 'failed'+ with self.lock:+ self.value = dict(running=False, fraction=0. if reason or stop.is_set() else 1., reason=reason)+++def available(installation=None):+ """The music model runs where video runs through CUDA (NVIDIA and AMD ROCm); not on Apple silicon yet."""+ try:+ if installation is None:+ from .comfy_bridge import installation+ _root, machine = installation()+ return machine.get('device_backend', 'cuda') != 'mps'+ except Exception: # noqa: BLE001 - no engine, no music+ return False+++class SpeechSpans:+ """Speech spans per editor source for 'Lower during speech'; one file at a time, off the request thread.++ The preview asks per source and gets pending until the spans are ready; the export asks for all of its+ sources at once and waits. Both read the same cached spans for the same file.+ """+ def __init__(self, root, *, resolve=None, model_path=None, compute=None):+ self.root = Path(root)+ self._resolve, self._model_path, self._compute = resolve, model_path, compute+ self.lock = threading.Lock()+ self.pending, self.failed, self.known = deque(), {}, {}+ self.worker = None++ def cache(self):+ return self.root / 'FreeVideo' / '.editor' / 'speech'++ def model_path(self):+ if self._model_path is not None:+ return Path(self._model_path)+ from .comfy_bridge import installation+ from .image_gen import image_pack+ from .speech import MODEL_FILE+ return image_pack.base(installation()[1]['model_root']) / MODEL_FILE++ def audio(self, key):+ if self._resolve is not None:+ return self._resolve(key)+ from .editor_export import resolve_source+ return resolve_source(key).get('audio')++ def compute(self, path, cancelled=None):+ if self._compute is not None:+ return self._compute(path)+ from .speech import speech_spans+ return speech_spans(path, self.model_path(), self.cache(), cancelled)++ def get(self, key):+ """{'spans': [...]} when known, {'state': 'pending'} while working, {'spans': None, 'reason': …} otherwise."""+ if not isinstance(key, str) or not re.fullmatch(r'(work|generated|import):[^\x00]{1,512}', key):+ raise ValueError('Invalid source')+ try:+ path = self.audio(key)+ except (FileNotFoundError, KeyError):+ return dict(spans=None, reason='source_missing')+ if path is None:+ return dict(spans=[])+ with self.lock:+ if key in self.known:+ return dict(spans=self.known[key])+ model = self.model_path()+ if not model.is_file():+ return dict(spans=None, reason='model_missing')+ from .speech import cached+ found = cached(path, model, self.cache())+ if found is not None:+ return dict(spans=found)+ with self.lock:+ if key in self.failed:+ return dict(spans=None, reason=self.failed[key])+ if key not in self.pending:+ if len(self.pending) >= MAX_QUIET:+ return dict(spans=None, reason='busy')+ self.pending.append(key)+ if self.worker is None or not self.worker.is_alive():+ self.worker = threading.Thread(target=self._run, name='freevideo-editor-speech', daemon=True)+ self.worker.start()+ return dict(state='pending')++ def _run(self):+ from .speech import SpeechError+ while True:+ with self.lock:+ if not self.pending:+ self.worker = None+ return+ key = self.pending[0]+ reason = found = None+ try:+ found = self.compute(self.audio(key))+ except SpeechError as error:+ reason = error.reason+ except Exception: # noqa: BLE001 - one unreadable source must not stop the others+ logging.getLogger(__name__).warning('Editor speech spans failed for %s.', key, exc_info=True)+ reason = 'failed'+ with self.lock:+ self.pending.popleft()+ if reason is not None:+ self.failed[key] = reason+ else:+ self.known[key] = found++ def spans(self, keys, cancelled=None):+ """Speech spans for each key the export needs; a source that cannot be read has none."""+ found = {}+ for key in keys:+ try:+ path = self.audio(key)+ found[key] = self.compute(path, cancelled) if path is not None else []+ except Exception as error: # noqa: BLE001 - no spans for that source, as in the preview+ if getattr(error, 'reason', None) == 'cancelled':+ raise+ logging.getLogger(__name__).warning('Editor export: no speech spans for %s (%s).', key,+ getattr(error, 'reason', type(error).__name__))+ return found+++def register():+ """Register the editing page's music endpoints once on the existing ComfyUI server."""+ import asyncio+ from aiohttp import web+ from server import PromptServer+ from .editor_export import _output_root+ server = PromptServer.instance+ if server is None or getattr(server, '_freevideo_editor_music', None) is not None:+ return+ queue = MusicQueue(_output_root(), server=server)+ pack = MusicPack()+ server._freevideo_editor_music = queue+ server._freevideo_editor_speech = speech = SpeechSpans(_output_root())+ headers = {'Cache-Control': 'no-store'}++ def response(value):+ return web.json_response({'job': value}, headers=headers)++ def refused(reason, status):+ return web.json_response({'reason': reason}, status=status, headers=headers)++ @server.routes.post('/freevideo/editor/music')+ async def start(request):+ try:+ body = await request.json()+ if not await asyncio.to_thread(available):+ return refused('unsupported', 409)+ return response(await asyncio.to_thread(queue.submit, body))+ except MusicBusy:+ return refused('busy', 503)+ except (ValueError, TypeError, KeyError):+ return refused('invalid_request', 400)++ @server.routes.get('/freevideo/editor/music')+ async def status(request):+ identity = request.query.get('id')+ if identity is None:+ supported = await asyncio.to_thread(available)+ try:+ ready = supported and (await asyncio.to_thread(pack.status))['ready']+ except Exception: # noqa: BLE001 - an unreadable pack reads as not downloaded+ ready = False+ # pace: seconds of composing per second of music here, from the last piece (null before the first).+ return web.json_response(dict(available=supported, ready=bool(ready), pace=queue.pace), headers=headers)+ try:+ return response(queue.get(identity))+ except KeyError:+ return refused('missing', 404)+ except ValueError:+ return refused('invalid_request', 400)++ @server.routes.post('/freevideo/editor/music/cancel')+ async def cancel(request):+ try:+ body = await request.json()+ if not isinstance(body, dict) or set(body) != {'id'}:+ raise ValueError('Invalid request')+ return response(queue.cancel(body['id']))+ except KeyError:+ return refused('missing', 404)+ except (ValueError, TypeError):+ return refused('invalid_request', 400)++ @server.routes.get('/freevideo/editor/speech')+ async def speech_spans(request):+ try:+ return web.json_response(await asyncio.to_thread(speech.get, request.query.get('source')), headers=headers)+ except ValueError:+ return refused('invalid_request', 400)++ @server.routes.get('/freevideo/editor/music-pack')+ async def pack_status(request):+ return web.json_response(await asyncio.to_thread(pack.status), headers=headers)++ @server.routes.post('/freevideo/editor/music-pack')+ async def pack_action(request):+ try:+ body = await request.json()+ if not isinstance(body, dict) or set(body) != {'action'} or body['action'] not in ('install', 'cancel'):+ raise ValueError('Invalid request')+ if body['action'] == 'cancel':+ pack.cancel()+ elif not await asyncio.to_thread(available):+ return refused('unsupported', 409)+ elif not await asyncio.to_thread(pack.start):+ return web.json_response(await asyncio.to_thread(pack.status), status=507, headers=headers)+ except (ValueError, TypeError):+ return refused('invalid_request', 400)+ return web.json_response(await asyncio.to_thread(pack.status), headers=headers)diff --git a/freevideo_engine/image_gen/image_models.json b/freevideo_engine/image_gen/image_models.jsonindex 797515d..86f7f86 100644--- a/freevideo_engine/image_gen/image_models.json+++ b/freevideo_engine/image_gen/image_models.json@@ -661,6 +661,29 @@"url": "https://www.modelscope.cn/models/ACE-Step/Ace-Step1.5/resolve/dd20ee00556360ae895e308efe9d11daf08fe9d2/vae/diffusion_pytorch_model.safetensors"}]+ },+ {+ "file": "music/silero-vad/model.onnx",+ "component": "music",+ "bytes": 2243022,+ "sha256": "a4a068cd6cf1ea8355b84327595838ca748ec29a25bc91fc82e6c299ccdc5808",+ "repo": "onnx-community/silero-vad",+ "revision": "e71cae966052b992a7eca6b17738916ce0eca4ec",+ "model_id": "silero-vad",+ "sources": [+ {+ "id": "official",+ "url": "https://huggingface.co/onnx-community/silero-vad/resolve/e71cae966052b992a7eca6b17738916ce0eca4ec/onnx/model.onnx"+ },+ {+ "id": "hf-mirror",+ "url": "https://hf-mirror.com/onnx-community/silero-vad/resolve/e71cae966052b992a7eca6b17738916ce0eca4ec/onnx/model.onnx"+ },+ {+ "id": "modelscope",+ "url": "https://www.modelscope.cn/models/onnx-community/silero-vad/resolve/2f080fcc38651e65b738157ad84c3c980beb08cc/onnx/model.onnx"+ }+ ]}]}diff --git a/freevideo_engine/image_gen/image_pack.py b/freevideo_engine/image_gen/image_pack.pyindex fe60e97..4b252f0 100644--- a/freevideo_engine/image_gen/image_pack.py+++ b/freevideo_engine/image_gen/image_pack.py@@ -229,7 +229,9 @@ SEGMENT_NOTICE_TEXT = (b'SAM 2.1 Hiera small, Copyright (c) Meta Platforms, Inc.b'https://github.com/facebookresearch/sam2\n')MUSIC_NOTICE_TEXT = (b'ACE-Step 1.5, Copyright (c) 2026 ACEStep. Licensed under the MIT License; see 'b'freevideo_engine/licenses/ACE-Step-MIT.txt.\nhttps://github.com/ace-step/ACE-Step-1.5\n'- b'Qwen3-Embedding-0.6B, Copyright (c) Alibaba Cloud. Licensed under Apache-2.0.\n')+ b'Qwen3-Embedding-0.6B, Copyright (c) Alibaba Cloud. Licensed under Apache-2.0.\n'+ b'Silero VAD, Copyright (c) 2020-present Silero Team. Licensed under the MIT License; see '+ b'freevideo_engine/licenses/Silero-VAD-MIT.txt.\nhttps://github.com/snakers4/silero-vad\n')NOTICES = {'image': ('NOTICE.txt', NOTICE_TEXT), 'segment': ('SAM2-NOTICE.txt', SEGMENT_NOTICE_TEXT),'music': ('MUSIC-NOTICE.txt', MUSIC_NOTICE_TEXT)}diff --git a/freevideo_engine/speech.py b/freevideo_engine/speech.pynew file mode 100644index 0000000..0a8fd04--- /dev/null+++ b/freevideo_engine/speech.py@@ -0,0 +1,279 @@+"""Where someone speaks in a sound file, so generated music can sit lower under dialogue.++Silero VAD (snakers4/silero-vad, MIT; the onnx-community export) runs on the CPU in torch: the 16 kHz+weights are read straight from the ONNX file, so no ONNX runtime is needed. The file is downloaded with the+music model. One pass over a file gives speech spans in seconds of that file; they are cached next to the+editor's other derived data, and the preview and the export read the same spans.+"""+from __future__ import annotations++import hashlib+import json+import os+from pathlib import Path+import threading++import numpy as np++MODEL_FILE = 'music/silero-vad/model.onnx'+VERSION = 1 # change when the spans for the same file would change+RATE = 16000+WINDOW, CONTEXT = 512, 64+ENTER, LEAVE = .5, .35 # Silero's own hysteresis+MIN_SPEECH, MIN_SILENCE, PAD = .25, .1, .03+# Music under dialogue stays down across short pauses between sentences instead of pumping.+JOIN_GAP = .6+LIMIT_SECONDS = 4 * 3600+++class SpeechError(Exception):+ def __init__(self, reason):+ super().__init__(reason)+ self.reason = reason+++# --- a small protobuf reader: only what an ONNX file's tensors need -----------------------------------------+def _varint(data, index):+ shift = value = 0+ while True:+ byte = data[index]+ index += 1+ value |= (byte & 0x7f) << shift+ if byte < 0x80:+ return value, index+ shift += 7+++def _fields(data):+ index = 0+ while index < len(data):+ key, index = _varint(data, index)+ number, wire = key >> 3, key & 7+ if wire == 0:+ value, index = _varint(data, index)+ elif wire == 1:+ value, index = data[index:index + 8], index + 8+ elif wire == 2:+ size, index = _varint(data, index)+ value, index = data[index:index + size], index + size+ elif wire == 5:+ value, index = data[index:index + 4], index + 4+ else:+ raise SpeechError('model')+ yield number, wire, value+++def _tensor(data):+ dims, kind, name, raw = [], None, '', None+ for number, wire, value in _fields(data):+ if number == 1:+ if wire == 2:+ index = 0+ while index < len(value):+ dim, index = _varint(value, index)+ dims.append(dim)+ else:+ dims.append(value)+ elif number == 2:+ kind = value+ elif number == 8:+ name = bytes(value).decode()+ elif number == 9:+ raw = value+ if kind != 1 or raw is None or len(raw) != 4 * int(np.prod(dims or [1])):+ return name, None+ return name, np.frombuffer(bytes(raw), '<f4').reshape(dims).copy()+++def _tensors(graph, found):+ for number, _, value in _fields(graph):+ if number == 5:+ name, array = _tensor(value)+ if array is not None:+ found[name] = array+ elif number == 1:+ for field, _, attribute in _fields(value):+ if field == 5:+ for part, _, inner in _fields(attribute):+ if part == 6:+ _tensors(inner, found)+ return found+++def onnx_weights(path):+ """The 16 kHz branch of the onnx-community Silero export, as numpy arrays named for _Model."""+ data = memoryview(Path(path).read_bytes())+ found = {}+ for number, _, value in _fields(data):+ if number == 7:+ _tensors(value, found)+ prefix = 'If_0_then_branch__Inline_0__'+ lstm = [array for name, array in found.items()+ if name.startswith(prefix) and name.endswith('_subg_96_sub_graph2') and array.ndim >= 2]+ try:+ weights = {'basis': found[prefix + 'stft.forward_basis_buffer']}+ for layer in range(4):+ weights[f'conv{layer}.weight'] = found[f'{prefix}encoder.{layer}.reparam_conv.weight']+ weights[f'conv{layer}.bias'] = found[f'{prefix}encoder.{layer}.reparam_conv.bias']+ weights['out.weight'] = found[prefix + 'decoder.decoder.2.weight']+ weights['out.bias'] = found[prefix + 'decoder.decoder.2.bias']+ w, r = [array[0] for array in lstm if array.shape == (1, 512, 128)]+ (b,) = [array[0] for array in lstm if array.shape == (1, 1024)]+ except (KeyError, ValueError):+ raise SpeechError('model') from None+ # ONNX gates are input, output, forget, cell; torch's are input, forget, cell, output.+ order = lambda x: np.concatenate([x[0:128], x[256:384], x[384:512], x[128:256]])+ weights.update({'lstm.weight_ih_l0': order(w), 'lstm.weight_hh_l0': order(r),+ 'lstm.bias_ih_l0': order(b[:512]), 'lstm.bias_hh_l0': order(b[512:])})+ if weights['basis'].shape != (258, 1, 256) or weights['conv0.weight'].shape != (128, 129, 3):+ raise SpeechError('model')+ return weights+++def _model(weights):+ import torch+ from torch import nn++ class Model(nn.Module):+ def __init__(self):+ super().__init__()+ self.register_buffer('basis', torch.zeros(258, 1, 256))+ self.conv0 = nn.Conv1d(129, 128, 3, padding=1)+ self.conv1 = nn.Conv1d(128, 64, 3, stride=2, padding=1)+ self.conv2 = nn.Conv1d(64, 64, 3, stride=2, padding=1)+ self.conv3 = nn.Conv1d(64, 128, 3, padding=1)+ self.lstm = nn.LSTM(128, 128, batch_first=True)+ self.out = nn.Conv1d(128, 1, 1)++ def forward(self, windows):+ """[n, 576] windows (64 context + 512 new samples) in order → [n] speech probabilities."""+ x = nn.functional.pad(windows, (0, 64), mode='reflect').unsqueeze(1)+ x = nn.functional.conv1d(x, self.basis, stride=128)+ x = torch.sqrt(x[:, :129] ** 2 + x[:, 129:] ** 2)+ for conv in (self.conv0, self.conv1, self.conv2, self.conv3):+ x = torch.relu(conv(x))+ x, _ = self.lstm(x.squeeze(-1).unsqueeze(0))+ x = torch.sigmoid(self.out(torch.relu(x[0]).unsqueeze(-1)))+ return x[:, 0, 0]++ model = Model()+ model.load_state_dict({name: torch.from_numpy(array) for name, array in weights.items()})+ return model.eval()+++_LOADED = {}+_LOCK = threading.Lock()+++def probabilities(samples, model_path):+ """float32 mono samples at 16 kHz → one speech probability per 512 samples (32 ms)."""+ import torch+ with _LOCK:+ key = str(model_path)+ if key not in _LOADED:+ _LOADED.clear()+ _LOADED[key] = _model(onnx_weights(model_path))+ model = _LOADED[key]+ samples = np.asarray(samples, np.float32)+ count = -(-len(samples) // WINDOW)+ if count == 0:+ return np.zeros(0, np.float32)+ padded = np.zeros(CONTEXT + count * WINDOW, np.float32)+ padded[CONTEXT:CONTEXT + len(samples)] = samples+ windows = np.lib.stride_tricks.sliding_window_view(padded, CONTEXT + WINDOW)[::WINDOW][:count]+ threads = torch.get_num_threads()+ try:+ torch.set_num_threads(min(threads, 4))+ with torch.inference_mode():+ return model(torch.from_numpy(np.ascontiguousarray(windows))).numpy()+ finally:+ torch.set_num_threads(threads)+++def spans(probs, duration):+ """Speech probabilities → merged [start, end] seconds, with Silero's hysteresis and padding."""+ step = WINDOW / RATE+ found, start, quiet = [], None, None+ for index, value in enumerate(probs):+ at = index * step+ if start is None:+ if value >= ENTER:+ start, quiet = at, None+ elif value < LEAVE:+ quiet = at if quiet is None else quiet+ if at + step - quiet >= MIN_SILENCE:+ found.append([start, quiet])+ start, quiet = None, None+ else:+ quiet = None+ if start is not None:+ found.append([start, len(probs) * step])+ joined = []+ for begin, end in found:+ if end - begin < MIN_SPEECH:+ continue+ begin, end = max(0., begin - PAD), min(duration, end + PAD)+ if joined and begin - joined[-1][1] <= JOIN_GAP:+ joined[-1][1] = max(joined[-1][1], end)+ else:+ joined.append([begin, end])+ return [[round(begin, 3), round(end, 3)] for begin, end in joined]+++def decode(path, cancelled=None):+ """The first audio stream of a media file as float32 mono at 16 kHz."""+ import av+ parts, total = [], 0+ try:+ with av.open(str(path)) as container:+ if not container.streams.audio:+ return np.zeros(0, np.float32)+ stream = container.streams.audio[0]+ resampler = av.AudioResampler(format='flt', layout='mono', rate=RATE)+ for frame in container.decode(stream):+ for out in resampler.resample(frame):+ part = out.to_ndarray().reshape(-1)+ parts.append(part)+ total += len(part)+ if total > LIMIT_SECONDS * RATE:+ raise SpeechError('too_long')+ if cancelled is not None and cancelled():+ raise SpeechError('cancelled')+ for out in resampler.resample(None):+ parts.append(out.to_ndarray().reshape(-1))+ except av.error.FFmpegError:+ raise SpeechError('unreadable') from None+ return np.concatenate(parts).astype(np.float32) if parts else np.zeros(0, np.float32)+++def _identity(path, model_path):+ stat, model = Path(path).stat(), Path(model_path).stat()+ raw = json.dumps([str(Path(path).resolve()), stat.st_size, stat.st_mtime_ns, model.st_size, VERSION])+ return hashlib.sha256(raw.encode()).hexdigest()[:32]+++def cached(path, model_path, cache):+ """The cached spans for this exact file and model, or None."""+ try:+ value = json.loads((Path(cache) / (_identity(path, model_path) + '.json')).read_text(encoding='utf-8'))+ return value['spans'] if isinstance(value, dict) and isinstance(value.get('spans'), list) else None+ except (OSError, ValueError, KeyError):+ return None+++def speech_spans(path, model_path, cache, cancelled=None):+ """Speech spans in seconds of `path`; computed once per file and model, then read from `cache`."""+ found = cached(path, model_path, cache)+ if found is not None:+ return found+ if not Path(model_path).is_file():+ raise SpeechError('model_missing')+ samples = decode(path, cancelled)+ found = spans(probabilities(samples, model_path), len(samples) / RATE)+ folder = Path(cache)+ folder.mkdir(parents=True, exist_ok=True)+ target = folder / (_identity(path, model_path) + '.json')+ temporary = target.with_name(f'{target.name}.{os.getpid()}.{threading.get_ident()}.tmp')+ temporary.write_text(json.dumps({'spans': found}), encoding='utf-8')+ os.replace(temporary, target)+ return founddiff --git a/web/editor.css b/web/editor.cssindex 8375feb..a6ea01d 100644--- a/web/editor.css+++ b/web/editor.css@@ -84,6 +84,14 @@.fv-editor .fv-ed-gen-retry:disabled{color:var(--fv-muted);cursor:default}.fv-ed-genbar .fv-request-assets{order:-2;flex:1 1 100%;min-width:0;font-size:12px;line-height:1.5}.fv-ed-genbar .fv-request-assets[hidden]{display:none}+/* A stretch of the music track: a style row and a mood row above the description. */+.fv-ed-genbar.fv-ed-genbar-music{flex-wrap:wrap}+.fv-ed-music-presets{order:-1;flex:1 1 100%;display:flex;flex-direction:column;gap:6px;min-width:0}+.fv-ed-music-presets[hidden]{display:none}+.fv-ed-music-row{display:flex;align-items:center;gap:8px;min-width:0}+.fv-ed-music-label{flex:none;min-width:2.5em;font-size:13px;color:var(--fv-muted)}+.fv-editor .fv-ed-music-presets .fv-ed-adjust-segmented{width:auto;flex:0 1 auto;min-width:0;flex-wrap:wrap}+.fv-editor .fv-ed-music-presets .fv-ed-adjust-segmented button{flex:none}.fv-editor:has(.fv-ed-gen-failure:not([hidden])){--fv-ed-generation-row:max-content}.fv-ed-genbar:has(.fv-ed-gen-failure:not([hidden])){flex-wrap:wrap;padding-top:10px;padding-bottom:10px}.fv-editor .fv-ed-gen-failure{display:block;flex:1 0 100%;min-width:0}diff --git a/web/editor.js b/web/editor.jsindex d1d3103..954043f 100644--- a/web/editor.js+++ b/web/editor.js@@ -21,9 +21,11 @@ import {createAdjustPicture} from './editor_adjust_picture.js';import {pictureSession} from './editor_picture_session.js';import {createPictureTools} from './editor_frame.js';import {openExportDialog} from './editor_export_dialog.js';-import {createAudioIntegration, handleAudioShortcut, AUDIO_SHORTCUTS} from './editor_audio.js';+import {createAudioIntegration, createSpeechSpans, handleAudioShortcut, AUDIO_SHORTCUTS} from './editor_audio.js';import {createSpeedComponent, readSpeedAudioSamples, handleSpeedKey, speedShortcuts} from './editor_speed.js';import {createEditorGenbar, loadEditorComponents, editorBlockStates, blockMessages} from './editor_genbar.js';+import {createEditorMusic} from './editor_music.js';+import {MAX_MUSIC_TEXT as MUSIC_TEXT_LIMIT} from './editor_music_model.js';import {createProgressConnection} from './progress_connection.js';import {createSelectionActions, deletionStatus, isMultiple, isTyping, playheadIds, removableGaps, removeGaps} from './editor_select.js';import {isolateDialogKeys, topDialog} from './dialog_keys.js';@@ -188,8 +190,9 @@ export function openEditor({api, app, from = null, panel = {}, components: givenconst t = (en, zh) => String(language()).toLowerCase().startsWith('zh') ? zh : en;const returnFocus = document.activeElement;const even = value => Math.max(2, Math.floor(value / 2) * 2);- const initial = () => model.emptyProject({width: even(panel.width || 1344), height: even(panel.height || 768), settings: panel.settings || null,- name: t('Untitled edit', '未命名剪辑')});+ // A new edit opens with an empty second sound track, where music is made.+ const initial = () => model.addAudioTrack(model.emptyProject({width: even(panel.width || 1344), height: even(panel.height || 768), settings: panel.settings || null,+ name: t('Untitled edit', '未命名剪辑')}));const state = createEditorState(initial());const resumeCanvasPointer = suspendCanvasPointer(app?.canvas);const element = el('dialog', 'fv-studio fv-editor'); element.setAttribute('aria-label', t('Edit', '剪辑视图')); element.tabIndex = -1;@@ -374,7 +377,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given}, {onStatus: () => {if (!queue.dirty) { discardOnSwitch = false; saveWarning.hidden = true; }}});- const player = createEditorPlayer(preview, transport, {state, sources: store.sources, api, t, error, readAudioSamples: panel.readAudioSamples || readSpeedAudioSamples, preservePreview: () => initializing, prepareAudio: () => !initializing});+ const player = createEditorPlayer(preview, transport, {state, sources: store.sources, api, t, error, readAudioSamples: panel.readAudioSamples || readSpeedAudioSamples, preservePreview: () => initializing, prepareAudio: () => !initializing, speech: createSpeechSpans(store.speech)});function selectionClips(selection = state.selection) {if (selection?.kind !== 'clip') return [];return state.record.tracks[0].clips.filter(clip => selection.ids.includes(clip.id) || selection.ids.includes(clip.link));@@ -515,6 +518,9 @@ export function openEditor({api, app, from = null, panel = {}, components: givenfunction completeSources() {return store.getSources(model.references(state.record).map(sourceKey)).then(() => { player.refresh(); tracks?.refresh?.(); });}+ // Music on the second sound track goes through the same bar; the bar shows a chosen piece's own job.+ const music = createEditorMusic({state, store, t, completeSources, onStatus: () => genbar?.update(barValue)});+ let barValue = null;const exportMatches = () => exportSnapshot && exportSnapshot.id === state.record.id && exportSnapshot.revision === state.record.revision && exportSnapshot.content === content(state.record);function explainExport() {if (initializing || switching) { notify(t('Opening the edit…', '正在打开…')); return true; }@@ -655,7 +661,7 @@ export function openEditor({api, app, from = null, panel = {}, components: giveninitialSelection = offerInitial = extendEntry = null;projectEpoch++; held.length = 0;player.pause(); for (const stop of jobWatchers.values()) stop(); jobWatchers.clear(); jobStatuses.clear(); progress.clear(); runProgress.clear(); failures.clear(); progressConnection?.reset();- mostRecentJob = null; submissionFailure = null; genbar?.update(null); queue.reset(record); stopExport?.(); stopExport = null; exportStatus = exportSnapshot = null; exportButton.textContent = t('Export', '导出');+ mostRecentJob = null; submissionFailure = null; music.reset(); genbar?.update(null); queue.reset(record); stopExport?.(); stopExport = null; exportStatus = exportSnapshot = null; exportButton.textContent = t('Export', '导出');suppressSave = true; try { state.replace(record); savedContent = content(record); } finally { suppressSave = false; }panel.onProjectLoaded?.(record);const width = tracksHost.clientWidth - 148; if (width > 0) state.setZoom(model.openingZoom(width, record));@@ -689,7 +695,7 @@ export function openEditor({api, app, from = null, panel = {}, components: givenforgetPointer();})); if (disposed) return;// A restored edit remains usable offline with its existing placeholders.- await completeSources().catch(value => error(value, null, undefined, 'sources')); if (disposed) return; recoverJobs();+ await completeSources().catch(value => error(value, null, undefined, 'sources')); if (disposed) return; recoverJobs(); music.recover();state.restoreView(remembered);// Also when generating is confirmed only after the edit has opened.offerInitial = () => { if (epoch === projectEpoch && !state.selection && state.record.tracks.every(track => !track.clips.length)) selectInitial(); };@@ -844,6 +850,7 @@ export function openEditor({api, app, from = null, panel = {}, components: giveninterrupted: clip => clip.pending === 'stopped' && !clip.source && !!clip.job&& (jobStatuses.get(clip.job)?.blocks || []).some(block => block.state === 'interrupted' && model.attemptClip(state.record, block.id)?.id === clip.id),selectedFailure() {+ const tune = music.failure(); if (tune) return tune;const clip = state.selection?.kind === 'clip' && state.record.tracks.flatMap(track => track.clips).find(clip => state.selection.ids.includes(clip.id) && clip.pending === 'failed');if (clip) return {id: clip.id, text: failures.text(clip.id)};return submissionFailure === JSON.stringify(state.selection) ? {id: 'submission', text: failures.text('submission')} : null;@@ -875,11 +882,13 @@ export function openEditor({api, app, from = null, panel = {}, components: given// The description the box shows for a choice of planned parts, and the parts it is written to; null for// any other choice.partText(selection = state.selection) {+ if (music.target(selection)) return music.text(selection);const clips = selectionClips(selection).sort((a, b) => a.start - b.start);return clips.length && clips.every(editablePart) ? {ids: clips.map(clip => clip.id), text: clips[0].text} : null;},describeParts(ids, text) {text = wellFormed(text);+ if (ids.length === 1 && music.target({kind: 'clip', ids})) return music.describe(ids[0], text);if (!ids.every(id => editablePart(model.find(state.record, id)))) return false;try { state.apply(t('Edit description', '修改说明'), record => model.describePlanned(record, ids, text)); return true; }catch (value) { if (value instanceof model.EditError) return false; throw value; }@@ -915,6 +924,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given}},previewRange: selection => genbar?.preview(selection),async estimate(selection) {+ if (music.target(selection)) return music.estimate(selection);const run = partRun(selection), range = selectionRange(run ? clipChoice(run) : selection); if (!range) return null;let seconds = 0;for (const count of model.rangeParts(state.record, {...range, ...capability})) { const value = capability.estimate(count); if (value == null) return null; seconds += value; }@@ -1012,6 +1022,11 @@ export function openEditor({api, app, from = null, panel = {}, components: givenasync press(text, {allowEmpty = false} = {}) {if (disposed || closing || initializing) return {say: t('Opening the edit…', '正在打开…')};text = wellFormed(text);+ // A stretch of the music track, or a piece of music there: music, with or without words.+ if (music.target()) {+ if (characters(text) > MUSIC_TEXT_LIMIT) return {say: textLongText(t, MUSIC_TEXT_LIMIT), focus: true};+ return music.press(text);+ }let selection = state.selection, said = null;if (actions.selectionHeld()) return {say: t('Queued', '排队中')};if (busySelection(selection)) return {say: t('This part is generating.', '这一段正在生成。')};@@ -1076,9 +1091,9 @@ export function openEditor({api, app, from = null, panel = {}, components: givenawait actions.generate({text: texts ? texts[run[0].id] : text, texts, selection, shown, allowEmpty});return {text, say: [said, used, busyBefore ? t('Queued. It starts after the part now generating.', '已排队,正在生成的那段完成后自动开始。') : null].filter(Boolean).join(' ') || null, queued: busyBefore};},- selectionRunning: () => busySelection(state.selection),+ selectionRunning: () => music.target() ? music.running() : busySelection(state.selection),// A choice over existing clips that Generate would move to an empty spot.- selectionMovable: () => !actions.supportedSelection() && !!emptySpot(state.selection),+ selectionMovable: () => !music.target() && !actions.supportedSelection() && !!emptySpot(state.selection),selectionSending() { const range = selectionRange(state.selection); return !!range && ([...sendingRanges].some(other => overlaps(other, range)) || [...sending].some(id => { const clip = model.find(state.record, id); return clip && overlaps(clip, range); })); },selectionHeld() { const ids = selectionClips(state.selection).map(clip => clip.id); return ids.length > 0 && ids.every(isHeld); },// Takes the selected parts out of the line; they stay as drafts.@@ -1088,6 +1103,7 @@ export function openEditor({api, app, from = null, panel = {}, components: givenrememberView();},async cancel() {+ if (music.target()) { await music.cancel(); return; }const range = selectionRange(state.selection);const own = range && state.record.tracks[0].clips.find(clip => clip.job && generating(clip)&& clip.start < range.start + range.frames && clip.start + clip.frames > range.start)?.job;@@ -1214,7 +1230,16 @@ export function openEditor({api, app, from = null, panel = {}, components: givenreturn running;}+ actions.music = {target: selection => music.target(selection), preset: selection => music.preset(selection), choose: value => music.choose(value),+ styles: music.styles, moods: music.moods, downloadBytes: music.downloadBytes};genbar = createEditorGenbar(genHost, {state, t, actions, text: ''}); settleService();+ {+ // What the editor gives the bar is the video job's; a chosen piece of music being made shows its own instead.+ const update = genbar.update;+ genbar.update = value => { barValue = value; update(music.barStatus() || value); };+ state.subscribe(event => { if (event.type === 'selection' || event.type === 'record') update(music.barStatus() || barValue); });+ music.refreshInfo();+ }const adjust = adjustPanel = createAdjustPanel(adjustHost, {state, t, actions});const pictureEdit = pictureSession(state, t);const notes = createBlockNotes({state, t, actions}); actions.blockNote = notes.decorate; actions.describedTrack = describedTrack;@@ -1315,7 +1340,7 @@ export function openEditor({api, app, from = null, panel = {}, components: givenconst next = content(state.record);if (!suppressSave && !initializing && next !== savedContent && exportStatus?.state === 'done') { exportStatus = null; exportButton.textContent = t('Export', '导出'); }if (!suppressSave && !initializing && next !== savedContent) { savedContent = next; queue.enqueue(state.record); }- completeSources().catch(value => error(value, null, undefined, 'sources')); recoverJobs();+ completeSources().catch(value => error(value, null, undefined, 'sources')); recoverJobs(); music.recover();}if (event.type === 'selection') { if (state.selection !== initialSelection) initialSelection = null; }if (event.type === 'record' || event.type === 'selection') { updateTop(); updateTransport(); pictureTools.update(); }@@ -1413,7 +1438,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given// A workflow without FreeVideo nodes came to the front: the editing view makes way (unsaved edits stay open).const collapse = () => { void handle.close(); };window.addEventListener('freevideo-collapse', collapse);- const handle = {element, state, actions, store, adjust, audio, notify,+ const handle = {element, state, actions, store, adjust, audio, music, notify,async switchView(view) {if (closing || disposed || view === 'editor') return;if (await handle.close() !== false) await onView(view);@@ -1440,7 +1465,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given}}await store.activateProject(null).catch(() => {});- rememberView(); disposed = true; guide?.dispose(); transportControls.dispose(); toolbarMore.dispose(); player.dispose();+ rememberView(); disposed = true; music.dispose(); guide?.dispose(); transportControls.dispose(); toolbarMore.dispose(); player.dispose();textTools.dispose(); adjust.dispose(); notes.dispose(); pictureEdit.dispose();audio.dispose(); headerFit.dispose();listeners.splice(0).forEach(remove => remove());@@ -1472,8 +1497,8 @@ export function openEditor({api, app, from = null, panel = {}, components: givenrecord = await store.byWork(id);if (!record) {const key = `work:${id}`; await store.getSources([key]); const metadata = store.sources.get(key), g = work.geometry || {};- record = model.emptyProject({width: even(metadata?.width || g.width || panel.width || 1344), height: even(metadata?.height || g.height || panel.height || 768),- settings: panel.settings || null, name: t('Untitled edit', '未命名剪辑')});+ record = model.addAudioTrack(model.emptyProject({width: even(metadata?.width || g.width || panel.width || 1344), height: even(metadata?.height || g.height || panel.height || 768),+ settings: panel.settings || null, name: t('Untitled edit', '未命名剪辑')}));record = model.insertSource(record, {source: {kind: 'work', id}, frames: metadata?.frames || g.frames || Math.round((g.seconds || 15) * 24),hasVideo: true, hasAudio: metadata?.has_audio ?? work.has_audio !== false, text: metadata?.text, at: 0});if (!metadata) store.cache({[key]: {video: mediaURL(api, work.video), audio: mediaURL(api, work.video), frames: g.frames,diff --git a/web/editor_audio.js b/web/editor_audio.jsindex b5b0fb3..40c79ac 100644--- a/web/editor_audio.js+++ b/web/editor_audio.js@@ -1,5 +1,5 @@import * as model from './editor_model.js';-import {row, slider, number, disabledReason} from './editor_adjust.js';+import {row, slider, number, toggle, disabledReason} from './editor_adjust.js';import * as sound from './editor_audio_model.js';import * as rules from './editor_audio_rules.js';import {replaceMode, attachReplacePicker} from './editor_audio_replace.js';@@ -127,6 +127,11 @@ export function createAudioPanel(host, {state, player, t = en => en, error = ()onChange: gain => { const ids = selectedAudio(state).map(clip => clip.id); editingVolume = false;apply(t('Volume', '音量'), record => ids.reduce((next, id) => model.setVolume(next, id, gain / 100), record)); },});+ // Generated music only: lowered while someone speaks in the other sound. On by default.+ const duck = toggle(t('Lower during speech', '对白时降低音量'), {+ onChange: on => apply(t('Lower during speech', '对白时降低音量'), record => sound.setDuck(record, selectedAudio(state).filter(sound.duckable).map(clip => clip.id), on)),+ });+ duck.classList.add('fv-ed-audio-duck'); form.append(duck); controls.push(duck);const fadeIn = input(form, 'fade-in', t('Fade in', '淡入'), 0, 5, 1 / 24, 'number', {onChange: seconds => apply(t('Fade in', '淡入'), record => sound.setFades(record, selectedAudio(state).map(clip => clip.id), 'fade_in', Math.round(seconds * 24))),});@@ -177,6 +182,9 @@ export function createAudioPanel(host, {state, player, t = en => en, error = ()const locked = selected.some(clip => sound.trackOf(state.record, clip.id).locked);const reason = disabledReason(ctx, {needs: 'audio'}) || (locked ? t('This track is locked', '这条轨道已锁定') : '');for (const field of [volume, fadeIn, fadeOut]) { field.disabled = !!reason; field.title = ''; field.input.title = ''; }+ const music = clips.filter(sound.duckable);+ duck.hidden = !music.length || music.length !== clips.length; duck.disabled = !!reason;+ if (!duck.hidden) { duck.value = music.every(clip => !!clip.duck); duck.input.indeterminate = music.some(clip => !!clip.duck) && !duck.value; }if (!editingVolume) {const gains = clips.map(clip => Math.round((clip.volume ?? 1) * 100)), mixed = gains.length > 1 && gains.some(gain => gain !== gains[0]);volume.value = mixed ? 100 : gains[0] ?? 100; setMixed(volume, mixed);@@ -314,10 +322,27 @@ export function attachAudioTracks(element, {state, t = en => en, actions = {}})// A decoded, bounded look-ahead path gives the sample envelope to Web Audio.// The unchanged media path remains active for records without audio edits.-export function createAudioPreview({state, sources, api, volumePreview = new Map(), readSamples = null, error = () => {}}) {+// Speech spans per source for "Lower during speech": asked once, polled while the server works them out.+// A source that cannot be read, or a server without the route, has no speech: nothing is lowered under it.+export function createSpeechSpans(fetchSpans, {retry = 1000} = {}) {+ const map = new Map(), asking = new Set(); let version = 0, disposed = false;+ const ask = key => {+ if (disposed || map.has(key) || asking.has(key)) return;+ asking.add(key);+ Promise.resolve().then(() => fetchSpans(key)).then(value => {+ asking.delete(key); if (disposed) return;+ if (value?.state === 'pending' || value?.reason === 'model_missing') { setTimeout(() => ask(key), value?.state === 'pending' ? retry : retry * 10); return; }+ map.set(key, Array.isArray(value?.spans) ? value.spans : []); version++;+ }).catch(() => { asking.delete(key); if (!disposed) { map.set(key, []); version++; } });+ };+ return {map, want(keys) { keys.forEach(ask); }, get version() { return version; }, dispose() { disposed = true; }};+}+export function createAudioPreview({state, sources, api, volumePreview = new Map(), readSamples = null, error = () => {}, speech = null}) {let context = null, generation = 0, timer = null, identity = null, pending = false, disposed = false;const cache = new Map(), nodes = new Set(); let origin = 0, cursor = 0, previousPosition = 0;- const edited = record => record.tracks.some(track => track.muted || track.solo || track.clips.some(clip => clip.fade_in || clip.fade_out || clip.crossfade));+ const ducking = (record, rate = 48000) => speech && sound.duckSources(record.tracks).length ? sound.duckRanges(record.tracks, speech.map, rate) : [];+ const edited = record => record.tracks.some(track => track.muted || track.solo || track.clips.some(clip => clip.fade_in || clip.fade_out || clip.crossfade))+ || ducking(record).length > 0;function stop() { generation++; clearTimeout(timer); timer = null; pending = false; identity = null; nodes.forEach(node => { try { node.stop(); } catch {} }); nodes.clear(); }async function decoded(clip) {const metadata = sources.get(sourceKey(clip.source)), url = mediaURL(api, metadata?.audio);@@ -343,7 +368,7 @@ export function createAudioPreview({state, sources, api, volumePreview = new Mapreturn raw(await decoded(clip), clip, offset, count);}async function chunk(record, at, count) {- const rate = context.sampleRate, total = [new Float64Array(count), new Float64Array(count)];+ const rate = context.sampleRate, total = [new Float64Array(count), new Float64Array(count)], lowered = ducking(record, rate);for (const track of record.tracks.filter(track => track.kind === 'audio')) {if (!sound.trackAudible(track, record.tracks)) continue;const block = [new Float64Array(count), new Float64Array(count)], extras = {};@@ -353,6 +378,7 @@ export function createAudioPreview({state, sources, api, volumePreview = new Mapconst a = Math.max(start, at), b = Math.min(stop, at + count);if (b <= a) continue;const data = await prepared(clip, a - start, b - a);+ const duck = clip.duck && lowered.length ? sound.duckGain(lowered, a, b - a, rate) : null;data.forEach((channel, c) => { for (let i = 0; i < channel.length; i++) {const local = a - start + i, length = stop - start;const prior = track.clips[track.clips.indexOf(clip) - 1], next = track.clips[track.clips.indexOf(clip) + 1];@@ -360,6 +386,7 @@ export function createAudioPreview({state, sources, api, volumePreview = new Maplet gain = clip.volume ?? 1;if (!clip.fade_in && (!prior?.source || prior.start + prior.frames !== clip.start || !prior.volume)) gain *= Math.min(1, local / (edge - 1));if (!clip.fade_out && (!next?.source || next.start !== clip.start + clip.frames || !next.volume)) gain *= Math.min(1, (length - 1 - local) / (edge - 1));+ if (duck) gain *= duck[i];block[c][a - at + i] = channel[i] * gain;}});}@@ -396,6 +423,7 @@ export function createAudioPreview({state, sources, api, volumePreview = new Map}return {render({record, position, playing, playbackRate = 1}) {+ speech?.want(sound.duckSources(record.tracks));if (!edited(record)) { if (identity) stop(); return false; }if (volumePreview.size) record = {...record, tracks: record.tracks.map(track => ({...track, clips: track.clips.map(clip => volumePreview.has(clip.id) ? {...clip, volume: volumePreview.get(clip.id)} : clip)}))};if (!playing) { stop(); previousPosition = position; return true; }@@ -404,7 +432,7 @@ export function createAudioPreview({state, sources, api, volumePreview = new Mapif (identity !== unavailable) { stop(); identity = unavailable; error(new model.EditError('audio_prepare')); }return true;}- const nextIdentity = JSON.stringify(record.tracks);+ const nextIdentity = JSON.stringify(record.tracks) + `:${speech?.version ?? 0}`;if (identity !== nextIdentity || Math.abs(position - previousPosition) > 3) {stop(); identity = nextIdentity;try { context ||= new (window.AudioContext || window.webkitAudioContext)({sampleRate: 48000}); context.resume().catch(error); }@@ -414,6 +442,6 @@ export function createAudioPreview({state, sources, api, volumePreview = new Map}previousPosition = position; return true;},- dispose() { disposed = true; stop(); cache.clear(); context?.close().catch(() => {}); },+ dispose() { disposed = true; stop(); cache.clear(); speech?.dispose(); context?.close().catch(() => {}); },};}diff --git a/web/editor_audio_model.js b/web/editor_audio_model.jsindex 754cd1f..5d8a52e 100644--- a/web/editor_audio_model.js+++ b/web/editor_audio_model.js@@ -16,6 +16,18 @@ export function audioClipFields(raw, kind) {if (typeof raw.detached_from !== 'string' || !/^c[1-9]\d{0,8}$/.test(raw.detached_from) || raw.detached_from === raw.id || raw.link) fail('link');fields.detached_from = raw.detached_from;}+ // A music piece's preset (what it was made with, prefilled when it is made again) and whether it is+ // lowered while the first sound track is heard.+ if (raw.music !== undefined) {+ const value = raw.music;+ if (kind !== 'audio' || !value || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).sort().join() !== 'mood,style'+ || ![value.style, value.mood].every(name => typeof name === 'string' && /^[a-z]{1,20}$/.test(name))) fail('audio');+ fields.music = {style: value.style, mood: value.mood};+ }+ if (raw.duck !== undefined) {+ if (kind !== 'audio' || typeof raw.duck !== 'boolean') fail('audio');+ fields.duck = raw.duck;+ }return fields;}export function audioTrackFields(raw) {@@ -97,6 +109,19 @@ export function setFades(record, ids, name, frames) {}return model.parse(next);}+// Lower during speech is a property of generated music only.+export const duckable = clip => !!clip?.music && !!clip.source;+export function setDuck(record, ids, on) {+ if (typeof on !== 'boolean' || !ids.length) fail('audio');+ editable(record, ids);+ const next = model.serialize(record);+ for (const id of ids) {+ const clip = model.find(next, id);+ if (trackOf(next, id)?.kind !== 'audio' || !duckable(clip)) fail('audio');+ clip.duck = on;+ }+ return model.parse(next);+}export function setCrossfade(record, id, frames) {if (!whole(frames)) fail('audio');const next = model.serialize(record), track = trackOf(next, id), right = model.find(next, id);@@ -205,3 +230,49 @@ export function applyTrackAudio(block, blockStart, track, tracks, {rate = 48000,output.forEach((channel, index) => { for (let i = 0; i < count; i++) channel[i] += added[index][i]; });return output;}+// Lower during speech: a clip with duck true sits DUCK_DB lower wherever another audible clip has speech,+// with DUCK_RAMP_SECONDS ramps just outside each stretch. freevideo_engine/editor_audio.py has the same rules.+export const DUCK_DB = -8, DUCK_RAMP_SECONDS = .3;+const changedSpeed = clip => (clip.speed ?? 1) !== 1 || !!clip.reverse;+// spans: Map or object of 'kind:id' → [[start, end] seconds]; ranges are timeline samples [first, last).+export function duckRanges(tracks, spans, rate = 48000, fps = 24) {+ const found = [], lookup = key => (spans instanceof Map ? spans.get(key) : spans?.[key]) || [];+ for (const track of tracks) {+ if (track.kind !== 'audio' || !trackAudible(track, tracks)) continue;+ for (const clip of track.clips) {+ if (!clip.source || clip.duck || (clip.volume ?? 1) === 0 || !clipAudible(clip, track, tracks)) continue;+ const first = samples(clip.start, rate, fps), last = samples(end(clip), rate, fps);+ const origin = changedSpeed(clip) || clip.speed_phase ? (clip.in * 2000 + (clip.speed_phase || 0) * 20) * rate / 48000 : sourceStart(clip, rate, fps);+ const speed = changedSpeed(clip) ? (clip.speed ?? 1) : 1;+ for (const [begin, stop] of lookup(`${clip.source.kind}:${clip.source.id}`)) {+ let a = (begin * rate - origin) / speed, b = (stop * rate - origin) / speed;+ if (clip.reverse) [a, b] = [(last - first) - b, (last - first) - a];+ a = Math.max(first, Math.floor(first + a + .5)); b = Math.min(last, Math.floor(first + b + .5));+ if (b > a) found.push([a, b]);+ }+ }+ }+ found.sort((x, y) => x[0] - y[0] || x[1] - y[1]);+ const gap = 2 * Math.max(1, Math.floor(DUCK_RAMP_SECONDS * rate + .5)), joined = [];+ for (const [a, b] of found) {+ if (joined.length && a <= joined.at(-1)[1] + gap) joined.at(-1)[1] = Math.max(joined.at(-1)[1], b);+ else joined.push([a, b]);+ }+ return joined;+}+export function duckGain(ranges, blockStart, count, rate = 48000) {+ const gain = new Float64Array(count).fill(1), ramp = Math.max(1, Math.floor(DUCK_RAMP_SECONDS * rate + .5)), low = 10 ** (DUCK_DB / 20);+ for (const [first, last] of ranges) {+ if (last + ramp <= blockStart || first - ramp >= blockStart + count) continue;+ for (let i = 0; i < count; i++) {+ const position = blockStart + i;+ const into = Math.max(0, Math.min(1, (position - (first - ramp)) / ramp)), outOf = Math.max(0, Math.min(1, ((last + ramp) - position) / ramp));+ gain[i] = Math.min(gain[i], 1 - (1 - low) * Math.min(into, outOf));+ }+ }+ return gain;+}+// The lowered clips' sources need speech spans from the clips they sit under.+export const duckSources = tracks => tracks.some(track => track.clips.some(clip => clip.duck && clip.source))+ ? [...new Set(tracks.filter(track => track.kind === 'audio').flatMap(track => track.clips)+ .filter(clip => clip.source && !clip.duck).map(clip => `${clip.source.kind}:${clip.source.id}`))] : [];diff --git a/web/editor_genbar.js b/web/editor_genbar.jsindex b6992f0..02a35e4 100644--- a/web/editor_genbar.js+++ b/web/editor_genbar.js@@ -6,6 +6,7 @@ import {createRequestAssets} from './request_assets.js';import {issueButton} from './issue_button.js';import {waitingText} from './chain_support.js';import {showEmptyPrompt} from './empty_prompt_notice.js';+import {segmented} from './editor_adjust.js';const el = (tag, cls, text) => { const e = document.createElement(tag); if (cls) e.className = cls; if (text != null) e.textContent = text; return e; };export const timeEstimate = (seconds, t) => seconds == null ? '' : seconds < 60? t(`about ${Math.max(1, Math.ceil(seconds))} s`, `约 ${Math.max(1, Math.ceil(seconds))} 秒`)@@ -65,6 +66,36 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {if (assets) host.prepend(assets.element);const firstFrame = createFirstFrameSwitch(host, {state, t, actions});description.after(firstFrame.element);+ // On the music track: a style and a mood above the description, each a row; the music's own first-use download+ // in the same line above the bar as the video's.+ const music = actions.music || null;+ const presets = music ? el('div', 'fv-ed-music-presets') : null;+ const musicAssets = music ? el('div', 'fv-request-assets fv-muted') : null;+ let musicBytes = null, musicBytesAt = 0;+ const size = bytes => bytes >= 1e9 ? (bytes / 1e9).toFixed(1) + '\u00a0GB' : Math.max(1, Math.round(bytes / 1e6)) + '\u00a0MB';+ const pickers = {};+ if (music) {+ for (const [key, label, options] of [['style', t('Style', '风格'), music.styles()], ['mood', t('Mood', '情绪'), music.moods()]]) {+ const row = el('div', 'fv-ed-music-row'), name = el('span', 'fv-ed-music-label', label);+ const picker = segmented(options.map(([value, text]) => ({value, label: text})), {onChange: value => { music.choose({[key]: value}); render(); }});+ picker.setAttribute('aria-label', label); pickers[key] = picker;+ row.append(name, picker); presets.append(row);+ }+ presets.hidden = true; musicAssets.hidden = true;+ description.before(presets); host.prepend(musicAssets);+ }+ async function showMusicAssets(shown) {+ if (!music) return;+ if (!shown) { musicAssets.hidden = true; return; }+ // Asked again at most every 5 s while a stretch of the music track is chosen.+ if (musicBytes === null || Date.now() - musicBytesAt > 5000) {+ musicBytesAt = Date.now();+ musicBytes = await music.downloadBytes();+ if (disposed) return;+ }+ musicAssets.textContent = musicBytes > 0 ? t(`Downloads before generating: music model ${size(musicBytes)}`, `生成前会先下载:配乐模型 ${size(musicBytes)}`) : '';+ musicAssets.hidden = !(musicBytes > 0) || host.dataset.mode !== 'selected' || !music.target();+ }const failure = el('div', 'fv-ed-gen-failure fv-result-links'), failureBar = el('div', 'fv-ed-failure-actions fv-actions');const failureReason = el('span', 'fv-ed-failure-reason'); failureReason.setAttribute('role', 'status');const report = el('button', 'fv-ed-failure-report fv-action fv-action-secondary', t('Download diagnostic report', '下载诊断报告')); report.type = 'button';@@ -119,9 +150,11 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {}failure.hidden = !selectedFailure;failureReason.textContent = selectedFailure?.text || '';- report.hidden = !selectedFailure;+ // A piece of music has no generation receipt to report.+ report.hidden = !selectedFailure || selectedFailure.report === false;const running = ['queued', 'running', 'cancelling'].includes(status?.state);- const selected = !!selection && (!!previewSelection || (actions.supportedSelection?.() ?? true));+ const musical = !!music?.target(selection);+ const selected = !!selection && (!!previewSelection || musical || (actions.supportedSelection?.() ?? true));// While a part generates, choosing another place offers Generate again; it is queued behind.// A press is being sent for the parts in `pressing`; only that same selection waits for it.const sending = pressing.has(JSON.stringify(state.selection ?? null)), starting = pressing.size > 0 && !running;@@ -134,13 +167,13 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {const stopped = ['failed', 'cancelled', 'interrupted'].includes(status?.state);host.dataset.mode = busy ? 'running' : selected ? 'selected' : 'idle';hint.hidden = !busy && !other;- // The description and Generate stay where they are while a part is made, greyed, so nothing around them moves.- input.hidden = generate.hidden = false;+ // The description and Generate stay where they are while a part is made, greyed, so nothing around them moves;+ // music being made shows only its progress and Cancel.+ input.hidden = false; generate.hidden = musical && busy;// A choice that will move to an empty spot can still be described first.const writable = selected || !previewSelection && !!selection && !!actions.selectionMovable?.();input.disabled = !writable || busy;- input.placeholder = writable ? t('Describe this part', '描述这一段')- : '';+ input.placeholder = !writable ? '' : musical ? t('Describe the music', '描述配乐') : t('Describe this part', '描述这一段');input.setAttribute('aria-label', t('Description', '描述'));generate.classList.toggle('fv-primary', selected && !busy);// A cancel already accepted is not offered again; the bar says it is stopping until the run ends.@@ -170,14 +203,26 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {hint.title = shown.map(([, text]) => text).join('');}firstFrame.update(busy);+ if (presets) {+ presets.hidden = !musical || !selected && !busy;+ if (!presets.hidden) {+ const value = music.preset(selection);+ for (const key of ['style', 'mood']) { pickers[key].value = value[key]; pickers[key].disabled = busy || !!previewSelection; }+ }+ host.classList.toggle('fv-ed-genbar-music', !presets.hidden);+ }references?.update(busy, busy ? undefined : actions.references.visible() || false);- generate.textContent = t('Generate', '生成');+ // A piece already made, failed or stopped is made again in place.+ const made = musical && !!music.target(selection)?.clip && music.target(selection).clip.pending !== 'draft';+ const verb = made ? t('Regenerate', '重新生成') : t('Generate', '生成');+ generate.textContent = verb;// Never disabled for a reason the press itself can explain or fix; only until the server says it can generate.generate.disabled = busy || !!previewSelection || !!actions.generationWaiting?.();resizeDescription();input.dispatchEvent(new Event('change'));generate.title = '';- if (assets) { if (selected && !busy) assets.refresh(); else assets.element.hidden = true; }+ if (assets) { if (selected && !busy && !musical) assets.refresh(); else assets.element.hidden = true; }+ showMusicAssets(selected && !busy && musical && !previewSelection);if (selected && !busy) {const key = JSON.stringify(selection);if (estimateCache?.key === key) generate.textContent += estimateCache.label;@@ -185,7 +230,7 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {if (disposed || current !== serial) return;const label = seconds == null ? '' : ' · ' + timeEstimate(seconds, t);estimateCache = {key, label};- generate.textContent = t('Generate', '生成') + label;+ generate.textContent = verb + label;}}function resizeDescription() {@@ -281,7 +326,7 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {if (!value && note.contains(retryService)) { retryService.remove(); note.hidden = true; }showService();}, getText() { expand.commit(); return input.value; }, update(value) { status = value; render(); }, preview(selection) { previewSelection = selection; render(); }, setText(value) { input.value = value || ''; state.setRangeText(input.value); if (bound && input.value !== bound.text) { dirty = true; written = input.value; } render(); },- dispose() { disposed = true; ++serial; clearTimeout(noteTimer); clearTimeout(refNoticeTimer); unsubscribe(); resize?.disconnect(); expand.dispose(); references?.dispose(); assets?.dispose(); }};+ dispose() { disposed = true; ++serial; clearTimeout(noteTimer); clearTimeout(refNoticeTimer); unsubscribe(); resize?.disconnect(); expand.dispose(); references?.dispose(); assets?.dispose(); Object.values(pickers).forEach(picker => picker.dispose()); }};}// A missing optional module leaves the editor's existing progress in place.diff --git a/web/editor_lifecycle.js b/web/editor_lifecycle.jsindex eed7b51..4054e09 100644--- a/web/editor_lifecycle.js+++ b/web/editor_lifecycle.js@@ -6,9 +6,11 @@ export function discardEmptyEdit(record, initial, {busy = false, hasMedia = falsif (busy || hasMedia || record.keep_empty || !untitled(record.name)|| record.tracks.some(track => track.clips.length) || record.replacements?.length) return false;// Compare settings, track controls and sparse project fields with initial().+ // An untouched empty second sound track does not make an edit differ from a new one.+ const bare = track => track.id === 'a2' && !track.clips.length && Object.keys(track).every(key => ['id', 'kind', 'clips'].includes(key));const content = value => {const {id, revision, serial, name, ...rest} = value;- return rest;+ return {...rest, tracks: rest.tracks.filter(track => !bare(track))};};return JSON.stringify(sorted(content(record))) === JSON.stringify(sorted(content(initial)));}diff --git a/web/editor_music.js b/web/editor_music.jsnew file mode 100644index 0000000..ff39c66--- /dev/null+++ b/web/editor_music.js@@ -0,0 +1,177 @@+// Music on the second sound track through the generation bar: a stretch chosen there (or a piece of music+// there) is what Generate makes, with the bar's own description box, a style and a mood. The pieces follow the+// music job's progress and take its file in place; the bar shows the job of the piece chosen.+import * as music from './editor_music_model.js';+import {EditError} from './editor_model.js';+import {timeEstimate} from './editor_genbar.js';++export const STYLE_NAMES = {cinematic: ['Cinematic', '电影感'], pop: ['Pop', '流行'], electronic: ['Electronic', '电子'],+ piano: ['Piano', '钢琴'], folk: ['Folk', '民谣'], ambient: ['Ambient', '氛围']};+export const MOOD_NAMES = {calm: ['Calm', '舒缓'], upbeat: ['Upbeat', '明快'], tense: ['Tense', '紧张'], warm: ['Warm', '温暖'], sad: ['Sad', '悲伤']};++// What the bar says about a piece's job: queued, downloading on first use, waiting for a video, or composing.+export function musicBarStatus(job, t, clock = Date.now) {+ if (!job || !['queued', 'running'].includes(job.state)) return null;+ let stage = '', time = '';+ if (job.state === 'queued') stage = t('Queued', '排队中');+ else if (job.phase === 'downloading') stage = t('Downloading files for generation', '正在下载生成所需文件');+ else if (job.phase === 'waiting') stage = t('Queued · waiting for the previous one', '排队中 · 等前一个完成');+ else if (job.phase === 'composing') {+ stage = t('Generating music', '正在生成配乐');+ // The server's elapsed time, carried forward between polls.+ const elapsed = Number.isFinite(job.elapsed) ? job.elapsed + Math.max(0, (clock() - (job.seen_at ?? clock())) / 1000) : null;+ if (Number.isFinite(job.estimate) && elapsed != null) time = elapsed > job.estimate * 1.2 + 5+ ? t('Taking longer than expected', '比预计久,仍在生成') : timeEstimate(Math.max(1, job.estimate - elapsed), t);+ } else stage = t('Preparing', '正在准备');+ return {state: 'running', blocks: [], music: true, words: {stage, ...(time ? {time} : {})}};+}++// A failed piece's sentence on the bar's failure line.+export function musicFailureText(reason, t) {+ return {+ music_download_failed: t('Download paused. Retry to resume.', '下载已暂停,重试可继续。'),+ disk_space: t('The disk is full. Free some space, then generate again.', '磁盘空间不足,请清理出空间后再生成。'),+ out_of_memory: t('Not enough memory to generate the music. Close other programs that use the graphics card, then generate again.', '生成配乐时内存不足,请关闭其他占用显卡的程序后重新生成。'),+ unsupported: t('Music cannot be generated on this computer yet.', '这台电脑暂不能生成配乐。'),+ missing: t('Generation stopped. Generate again.', '生成已中断,请重新生成。'),+ }[reason] || t('The music could not be generated. Generate again.', '无法生成配乐,请重新生成。');+}++export function createEditorMusic({state, store, t, completeSources = async () => {}, onStatus = () => {}, clock = Date.now, random = Math.random}) {+ const statuses = new Map(), watching = new Map();+ let info = null, infoAsked = null, chosen = null, chosenFor = null, disposed = false;+ const target = (selection = state.selection) => music.musicTarget(state.record, selection);+ const refreshInfo = () => infoAsked ??= Promise.resolve().then(() => store.musicInfo()).then(value => { info = value || {available: false}; return info; })+ .catch(() => (info = null)).finally(() => { infoAsked = null; onStatus(); });+ // The style and mood the bar shows: the chosen piece's own, else the edit's last choice; a pick on the bar holds+ // until the choice moves elsewhere.+ function preset(selection = state.selection) {+ const key = JSON.stringify(selection ?? null);+ if (chosen && chosenFor === key) return chosen;+ const found = target(selection);+ return found?.clip?.music && music.MUSIC_STYLES.includes(found.clip.music.style) && music.MUSIC_MOODS.includes(found.clip.music.mood)+ ? {...found.clip.music} : music.lastPreset(state.record);+ }+ function choose(value) {+ chosen = {...preset(), ...value}; chosenFor = JSON.stringify(state.selection ?? null);+ }+ // A piece removed or changed meanwhile is left as it is.+ const settle = change => { try { state.settle(change); } catch (value) { if (!(value instanceof EditError)) throw value; } };+ function deliver(id, jobId, job) {+ if (disposed || !job) return;+ statuses.set(jobId, {...job, seen_at: clock()});+ if (job.state === 'done' && job.result) {+ stopWatching(jobId);+ settle(record => music.finishMusic(record, id, jobId, {source: job.result.source, frames: job.result.frames}));+ completeSources().catch(() => {});+ // The piece just made sets this machine's pace, so the next Generate can show its estimate.+ refreshInfo();+ } else if (['failed', 'cancelled'].includes(job.state)) {+ stopWatching(jobId);+ settle(record => music.failMusic(record, id, jobId, job.state === 'cancelled' ? 'cancelled' : job.reason || 'music_failed'));+ } else settle(record => music.musicProgress(record, id, jobId, job.state));+ onStatus();+ }+ const starting = new Set(); let pressing = false;+ function stopWatching(jobId) { watching.get(jobId)?.(); watching.delete(jobId); }+ function watch(id, jobId) {+ if (watching.has(jobId)) return;+ watching.set(jobId, store.watchMusic(jobId, job => deliver(id, jobId, job), () => deliver(id, jobId, {id: jobId, state: 'failed', reason: 'missing'})));+ }+ async function begin(id, words) {+ state.select({kind: 'clip', ids: [id]});+ chosen = null;+ let job;+ try { job = await store.startMusic(music.musicRequest(state.record, id, {seed: Math.floor(random() * 2 ** 32)})); }+ catch (value) {+ settle(record => music.failMusic(record, id, undefined, value?.reason === 'unsupported' ? 'unsupported' : 'music_failed'));+ onStatus();+ return {say: musicFailureText(value?.reason === 'unsupported' ? 'unsupported' : 'music_failed', t)};+ }+ if (!job?.id) { settle(record => music.failMusic(record, id, undefined)); onStatus(); return {say: musicFailureText('music_failed', t)}; }+ settle(record => music.assignMusicJob(record, id, job.id));+ statuses.set(job.id, {...job, seen_at: clock()});+ watch(id, job.id); onStatus();+ return {text: words};+ }+ const api = {+ target, preset, choose, refreshInfo,+ get info() { return info; },+ styles: () => music.MUSIC_STYLES.map(id => [id, t(...STYLE_NAMES[id])]),+ moods: () => music.MUSIC_MOODS.map(id => [id, t(...MOOD_NAMES[id])]),+ // The chosen piece is being made.+ running(selection = state.selection) {+ const found = target(selection);+ return !!found?.clip && (starting.has(found.clip.id) || ['queued', 'running'].includes(found.clip.pending));+ },+ // The bar's status for the chosen piece's job; null when it has none running.+ barStatus(selection = state.selection) {+ const found = target(selection), job = found?.clip?.job && statuses.get(found.clip.job);+ return job && ['queued', 'running'].includes(found.clip.pending) ? musicBarStatus(job, t, clock) : null;+ },+ // The words the box shows for a piece of music; null for a stretch not made yet.+ text(selection = state.selection) {+ const found = target(selection);+ return found?.clip && !api.running(selection) ? {ids: [found.clip.id], text: found.clip.text} : null;+ },+ describe(id, text) {+ const found = target({kind: 'clip', ids: [id]});+ if (!found?.clip || api.running({kind: 'clip', ids: [id]}) || found.clip.text === text) return false;+ state.apply(t('Edit description', '修改说明'), record => {+ const next = JSON.parse(JSON.stringify(record));+ next.tracks.find(track => track.id === music.MUSIC_TRACK).clips.find(clip => clip.id === id).text = text;+ return next;+ });+ return true;+ },+ failure(selection = state.selection) {+ const found = target(selection);+ return found?.clip?.pending === 'failed' ? {id: found.clip.id, text: musicFailureText(found.clip.reason, t), report: false} : null;+ },+ // Seconds the piece will take: this machine's pace from its last piece; unknown before the first.+ estimate(selection = state.selection) {+ const found = target(selection);+ return found && Number.isFinite(info?.pace) ? info.pace * found.frames / 24 : null;+ },+ // First-use download for the bar's "Downloads before generating" line: bytes still to fetch, or 0.+ async downloadBytes() {+ const pack = await Promise.resolve().then(() => store.musicPack()).catch(() => null);+ return pack && !pack.ready && !pack.running ? pack.bytes || 0 : 0;+ },+ // Generate on a stretch of the music track, or again on a piece there. {say} answers the press.+ // A second press while the first is still checking or starting its job is the same press.+ async press(text) {+ const found = target();+ if (!found) return null;+ if (pressing || api.running()) return {say: t('This part is generating.', '这一段正在生成。')};+ pressing = true;+ let id = found.id, words;+ try {+ const current = info || await refreshInfo();+ if (current && current.available === false) return {say: musicFailureText('unsupported', t)};+ const pack = await Promise.resolve().then(() => store.musicPack()).catch(() => null);+ if (pack && !pack.ready && pack.disk_ok === false) return {say: musicFailureText('disk_space', t)};+ const pick = preset(); words = text.trim();+ state.apply(t('Generate', '生成'), record => {+ const next = music.rememberPreset(record, pick);+ if (id) return music.remakeMusic(next, id, {text: words, ...pick});+ const planned = music.planMusic(next, {start: found.start, frames: found.frames, text: words, ...pick});+ id = planned.id; return planned.record;+ }, {selectionAfter: null});+ starting.add(id);+ } finally { pressing = false; }+ try { return await begin(id, words); } finally { starting.delete(id); }+ },+ async cancel(selection = state.selection) {+ const found = target(selection);+ if (!found?.clip?.job || !api.running(selection)) return false;+ deliver(found.clip.id, found.clip.job, await store.cancelMusic(found.clip.job).then(value => value?.job || value));+ return true;+ },+ // Pieces still waiting for their job when the edit opens again.+ recover() { for (const {id, job} of music.musicJobs(state.record)) watch(id, job); },+ reset() { for (const jobId of [...watching.keys()]) stopWatching(jobId); statuses.clear(); chosen = null; },+ dispose() { disposed = true; api.reset(); },+ };+ return api;+}diff --git a/web/editor_music_model.js b/web/editor_music_model.jsnew file mode 100644index 0000000..2f020e9--- /dev/null+++ b/web/editor_music_model.js@@ -0,0 +1,132 @@+// A piece of music planned on a stretch of the second sound track, made by the music job and laid back in place.+// Pure edits on the project record; the generation bar decides when to call them.+import * as model from './editor_model.js';+import {FPS} from './editor_model.js';++export const MUSIC_TRACK = 'a2';+// The longest piece the music model makes in one go.+export const MAX_MUSIC_FRAMES = 600 * FPS;+// The longest description the music job takes.+export const MAX_MUSIC_TEXT = 512;+// The presets the server knows; the user's own words go instead when given.+export const MUSIC_STYLES = ['cinematic', 'pop', 'electronic', 'piano', 'folk', 'ambient'];+export const MUSIC_MOODS = ['calm', 'upbeat', 'tense', 'warm', 'sad'];+const fail = code => { throw new model.EditError(code); };+const end = clip => clip.start + clip.frames;+const lane = record => record.tracks.find(track => track.id === MUSIC_TRACK) || null;++// The music track's planned or running pieces: no file yet, waiting for (or made by) a music job.+export const isMusicPart = (record, clip) => !!clip && !clip.source && clip.pending !== null+ && !!lane(record)?.clips.some(other => other.id === clip.id);+// A finished piece the music job made: its file sits in the music folder.+export const isGeneratedMusic = clip => clip?.source?.kind === 'generated' && /^freevideo-music\/[a-z]+(-[a-z]+)?-[0-9a-f]{32}\.mp4$/.test(clip.source.id);++export const DEFAULT_PRESET = {style: 'cinematic', mood: 'calm'};+const preset = value => MUSIC_STYLES.includes(value?.style) && MUSIC_MOODS.includes(value?.mood) ? {style: value.style, mood: value.mood} : null;+// The style and mood last chosen in this edit; the default pair before any.+export const lastPreset = record => preset(record.settings?.music) || {...DEFAULT_PRESET};+// Remember the pair for the next piece in this edit.+export function rememberPreset(record, value) {+ const chosen = preset(value) || fail('preset');+ if (JSON.stringify(record.settings?.music) === JSON.stringify(chosen)) return model.parse(record);+ const next = model.serialize(record);+ next.settings = {...(next.settings || {}), music: chosen};+ return model.parse(next);+}+// What a selection on the music track asks for: an empty stretch drawn or clicked there, or one piece of music+// there (made again in place). Null for anything else.+export function musicTarget(record, selection) {+ const track = lane(record);+ if (!track || !selection) return null;+ if (selection.kind === 'range' && selection.tracks?.length === 1 && selection.tracks[0] === MUSIC_TRACK+ || selection.kind === 'gap' && selection.track === MUSIC_TRACK) {+ const frames = Math.min(selection.frames, MAX_MUSIC_FRAMES);+ if (track.clips.some(clip => clip.start < selection.start + frames && end(clip) > selection.start)) return null;+ return {start: selection.start, frames};+ }+ if (selection.kind !== 'clip' || selection.ids.length !== 1) return null;+ const clip = track.clips.find(clip => clip.id === selection.ids[0]);+ if (!clip || !(isMusicPart(record, clip) || isGeneratedMusic(clip))) return null;+ return {start: clip.start, frames: clip.frames, id: clip.id, clip};+}++// A drag on the music track: the stretch from start for frames, which must be empty there. The track is added when missing.+export function planMusic(record, {start, frames, text = '', style, mood}) {+ if (!Number.isSafeInteger(start) || start < 0 || !Number.isSafeInteger(frames) || frames < 1) fail('range');+ if (frames > MAX_MUSIC_FRAMES) fail('too_long');+ if (typeof text !== 'string') fail('text');+ const chosen = style === undefined && mood === undefined ? {...DEFAULT_PRESET} : preset({style, mood}) || fail('preset');+ const next = model.serialize(lane(record) ? record : model.addAudioTrack(record)), track = next.tracks.find(track => track.id === MUSIC_TRACK);+ if (track.locked) fail('locked');+ if (track.clips.some(clip => clip.start < start + frames && end(clip) > start)) fail('overlap');+ const id = `c${++next.serial}`;+ track.clips.push({id, start, frames, in: 0, length: frames, source: null, link: null, text, part: null, pending: 'draft',+ reason: null, job: null, extend: false, versions: [], volume: 1, music: chosen, duck: true});+ track.clips.sort((a, b) => a.start - b.start);+ return {record: model.parse(next), id};+}+// Made again in place: a finished, failed or stopped piece becomes a planned one over the same stretch, with the+// words and preset given; its volume, fades and lowering stay. One undo brings the previous piece back.+export function remakeMusic(record, id, {text = '', style, mood}) {+ if (typeof text !== 'string') fail('text');+ const chosen = preset({style, mood}) || fail('preset');+ const next = model.serialize(record), clip = next.tracks.find(track => track.id === MUSIC_TRACK)?.clips.find(clip => clip.id === id);+ if (!clip || !(isGeneratedMusic(clip) || !clip.source && clip.pending !== null)) fail('missing');+ if (next.tracks.find(track => track.id === MUSIC_TRACK).locked) fail('locked');+ if (['queued', 'running'].includes(clip.pending)) fail('busy');+ Object.assign(clip, {source: null, in: 0, length: clip.frames, text, pending: 'draft', reason: null, job: null, versions: [], music: chosen});+ delete clip.splice;+ return model.parse(next);+}++// What the music job is asked for: the piece's length, its words when there are any, and its preset.+export function musicRequest(record, id, {seed} = {}) {+ const clip = model.find(record, id);+ if (!isMusicPart(record, clip)) fail('missing');+ const body = {frames: clip.frames, ...(clip.music || DEFAULT_PRESET)};+ if (clip.text.trim()) body.text = clip.text.trim();+ if (Number.isSafeInteger(seed)) body.seed = seed;+ return body;+}++function change(record, id, job, edit) {+ const next = model.serialize(record), clip = next.tracks.find(track => track.id === MUSIC_TRACK)?.clips.find(clip => clip.id === id);+ if (!clip || clip.source || clip.pending === null) return null;+ if (job !== undefined && clip.job !== job) return null;+ edit(clip);+ return model.parse(next);+}+// The job was accepted: the piece waits for it.+export function assignMusicJob(record, id, job) {+ if (typeof job !== 'string' || !/^m-[0-9a-f]{32}$/.test(job)) fail('job');+ return change(record, id, undefined, clip => { clip.job = job; clip.pending = 'queued'; clip.reason = null; }) ?? fail('missing');+}+// What the job says while it runs; a piece already finished, removed or given to another job stays as it is.+export function musicProgress(record, id, job, state) {+ if (!['queued', 'running'].includes(state)) return model.parse(record);+ return change(record, id, job, clip => { clip.pending = state; }) ?? model.parse(record);+}+// The file is made: it goes into the planned stretch. A shorter file shortens the piece; nothing else moves.+export function finishMusic(record, id, job, {source, frames}) {+ const [kind, path] = typeof source === 'string' ? [source.slice(0, source.indexOf(':')), source.slice(source.indexOf(':') + 1)] : [];+ if (kind !== 'generated' || !path || !Number.isSafeInteger(frames) || frames < 1) fail('source');+ return change(record, id, job, clip => {+ Object.assign(clip, {source: {kind, id: path}, in: 0, length: frames, frames: Math.min(clip.frames, frames), pending: null, reason: null, job: null});+ }) ?? model.parse(record);+}+// Failed or stopped: the piece stays where it was drawn, with its reason, ready to try again.+export function failMusic(record, id, job, reason = 'music_failed') {+ const state = reason === 'cancelled' ? 'stopped' : 'failed';+ const code = typeof reason === 'string' && /^[a-z_]{1,80}$/.test(reason) ? reason : 'music_failed';+ return change(record, id, job, clip => { clip.pending = state; clip.reason = state === 'failed' ? code : null; }) ?? model.parse(record);+}+// Try again: back to planned, with the same stretch and description.+export function retryMusic(record, id) {+ return change(record, id, undefined, clip => {+ if (['queued', 'running'].includes(clip.pending)) fail('busy');+ clip.pending = 'draft'; clip.reason = null; clip.job = null;+ }) ?? fail('missing');+}+// The pieces still waiting for a job after the page opens again, to watch or to mark failed.+export const musicJobs = record => (lane(record)?.clips || []).filter(clip => !clip.source && clip.job && ['queued', 'running'].includes(clip.pending))+ .map(clip => ({id: clip.id, job: clip.job}));diff --git a/web/editor_player.js b/web/editor_player.jsindex 0524d7c..3a7cb78 100644--- a/web/editor_player.js+++ b/web/editor_player.js@@ -27,7 +27,7 @@ export function selectAudioChain(chains, track, clip, position, volumePreview =&& !c.clips.some(id => volumePreview.has(id) && volumePreview.get(id) !== 1));return chain ? {chain, seconds: (position - chain.start) / 24 + (chain.offset || 0)} : null;}-export function createEditorPlayer(host, transport, {state, sources, api, t, error = () => {}, readAudioSamples = null, preservePreview = () => false, prepareAudio = () => true}) {+export function createEditorPlayer(host, transport, {state, sources, api, t, error = () => {}, readAudioSamples = null, preservePreview = () => false, prepareAudio = () => true, speech = null}) {const frame = el('div', 'fv-ed-picture'); frame.setAttribute('aria-label', t('Video preview', '视频预览'));const empty = el('div', 'fv-ed-empty-preview'), pictureIcon = document.createElementNS('http://www.w3.org/2000/svg', 'svg');pictureIcon.setAttribute('viewBox', '0 0 24 24'); pictureIcon.setAttribute('aria-hidden', 'true');@@ -54,7 +54,7 @@ export function createEditorPlayer(host, transport, {state, sources, api, t, errconst chainSeconds = (chain, position) => Math.max(0, (position - chain.start) / 24 + (chain.offset || 0));const playback = createPlaybackClock();const gains = new WeakMap(), volumePreview = new Map();- const audioPreview = createAudioPreview({state, sources, api, volumePreview, readSamples: readAudioSamples, error});+ const audioPreview = createAudioPreview({state, sources, api, volumePreview, readSamples: readAudioSamples, error, speech});const clipVolume = clip => volumePreview.get(clip?.id) ?? clip?.volume ?? 1;function volume(media, value) {// HTMLMediaElement stops at 1. A GainNode implements the record's 0–2 range.diff --git a/web/editor_store.js b/web/editor_store.jsindex 9590f9b..c95a28a 100644--- a/web/editor_store.js+++ b/web/editor_store.js@@ -514,6 +514,33 @@ export function createEditorStore(api, {t = en => en, notify = () => {}, onGenerinstallEars: () => json('/freevideo/editor/subtitle-pack', post({action: 'install', language: true})),cancelEarsPack: () => json('/freevideo/editor/subtitle-pack', post({action: 'cancel'})),watchEarsPack(fn) { return watch(() => json('/freevideo/editor/subtitle-pack'), fn); },+ // Music for a stretch of the sound track. {available, ready}; null where the server is older.+ musicInfo: () => json('/freevideo/editor/music'),+ // body: {frames, text?} or {frames, style, mood}, optional seed. Refusals keep the server's reason.+ async startMusic(body) {+ const response = await request('/freevideo/editor/music', post(body));+ const value = await response.json().catch(() => null);+ if (!response.ok) throw Object.assign(errorFor(response), {reason: value?.reason || (response.status === 404 ? 'unsupported' : 'failed')});+ return value?.job || null;+ },+ // A job the server no longer knows (it was restarted) reads as failed, so the planned piece does not wait forever.+ watchMusic(id, fn, lost) {+ return watch(async () => (await json('/freevideo/editor/music?' + new URLSearchParams({id}), undefined, {job: {id, state: 'failed', reason: 'missing'}}))?.job || null, fn, lost);+ },+ cancelMusic: id => json('/freevideo/editor/music/cancel', post({id})),+ // The music model is downloaded on first use, with the download source chosen in Settings.+ musicPack: () => json('/freevideo/editor/music-pack'),+ // Speech spans of one source in seconds, for "Lower during speech": {spans}, {state: 'pending'} or {spans: null, reason}.+ speech: key => json('/freevideo/editor/speech?' + new URLSearchParams({source: key})),+ async installMusic() {+ const response = await request('/freevideo/editor/music-pack', post({action: 'install'}));+ const value = await response.json().catch(() => null);+ // 507: not enough room on the disk; the state that comes back says how much is needed.+ if (!response.ok && response.status !== 507) throw Object.assign(errorFor(response), {reason: value?.reason || 'failed'});+ return value;+ },+ cancelMusicPack: () => json('/freevideo/editor/music-pack', post({action: 'cancel'})),+ watchMusicPack(fn) { return watch(() => json('/freevideo/editor/music-pack'), fn); },dispose() { disposed = true; activateProject(null).catch(() => {}); session.dispose(); for (const poller of pollers) poller.dispose(); pollers.clear(); watchers.clear();for (const upload of [...uploads]) { upload.release(); upload.xhr.abort(); upload.reject(new Error('editor_closed')); } },};diff --git a/web/editor_tracks.js b/web/editor_tracks.jsindex e02d216..af51c30 100644--- a/web/editor_tracks.js+++ b/web/editor_tracks.js@@ -13,6 +13,9 @@ import {attachAudioTracks} from './editor_audio.js';import {seconds as secondsText} from './editor_audio_notes.js';import {renderSpeedBadge} from './editor_speed.js';import {speedEdited} from './editor_speed_model.js';+import {MUSIC_TRACK, MAX_MUSIC_FRAMES} from './editor_music_model.js';+// A range drawn on the music track alone: music for that stretch.+const musicRange = selection => selection?.kind === 'range' && selection.tracks?.length === 1 && selection.tracks[0] === MUSIC_TRACK;export const SOURCE_TYPE = 'application/x-freevideo-source';export const SNAP_DISTANCE = 8;@@ -280,11 +283,13 @@ export function createTracks(host, {state, sources = {}, t = en => en, actions =}// A range cannot be drawn longer than one video may be; `fromEnd` keeps its end where it is.function capRange(selection, fromEnd = false, continues = false) {- const most = model.maxRangeFrames(state.record, {...actions.generationLimits?.(), continues});+ const most = musicRange(selection) ? MAX_MUSIC_FRAMES : model.maxRangeFrames(state.record, {...actions.generationLimits?.(), continues});if (selection.frames <= most) return selection;return fromEnd ? {...selection, start: selection.start + selection.frames - most, frames: most} : {...selection, frames: most};}function rangeParts(selection, record = state.record) {+ // A stretch of the music track is one piece, never split into video parts.+ if (musicRange(selection)) return [selection.frames];return model.rangeParts(record, {...selection, ...actions.generationLimits?.()});}function boundaryLine(frame, index) {@@ -806,7 +811,8 @@ export function createTracks(host, {state, sources = {}, t = en => en, actions =const aligned = snap(frame, points, state.zoom);// Where it was pressed, drawn to a clip edge, marker or the playhead nearby.gesture.from ??= snap(gesture.anchor, points, state.zoom).frame;- const drawn = rangeParameters(gesture.from, aligned.frame);+ // Drawn on the music track, the range is that track's alone: music for that stretch.+ const drawn = rangeParameters(gesture.from, aligned.frame, gesture.track === MUSIC_TRACK ? [MUSIC_TRACK] : undefined);// Drawn from a clip's end, the range starts exactly there.if (gesture.flush != null && aligned.frame > gesture.flush) Object.assign(drawn, {start: gesture.flush,frames: Math.max(2 * model.FPS, Math.round((aligned.frame - gesture.flush) / model.FPS) * model.FPS)});