Skip to content

Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.

Add voice change: recognise the lines of a clip and say the edited ones again in the original...

Bowen Xue5df13f7736e8
diff --git a/freevideo_engine/agent/errors.py b/freevideo_engine/agent/errors.py
index 8f11a97..1df7d39 100644
--- a/freevideo_engine/agent/errors.py
+++ b/freevideo_engine/agent/errors.py
@@ -125,6 +125,14 @@ ERRORS.update({
'Tell the user the model needs a network connection, then submit the job again with a new ref.'),
'stems_failed': (409, True, 'Splitting the sound did not finish.',
'Read the project, then submit the job again with a new ref; if it fails again, leave the sound as it is.'),
+ 'voice_failed': (409, True, 'Changing the voice did not finish.',
+ 'Read the project, then submit the job again with a new ref; if it fails again, leave the sound as it is.'),
+ 'voice_model_missing': (409, False, 'The dubbing model is not ready on this machine.',
+ 'Tell the user to install the dubbing model first; do not retry.'),
+ 'no_speech': (409, False, 'No speech was found in the selection.',
+ 'Do not retry; choose a stretch of sound with someone speaking.'),
+ 'shorten_unavailable': (409, False, 'Lines cannot be shortened here yet.',
+ 'Write the shorter line yourself and send it with the other lines.'),
'clip_source_unusable': (409, False, 'A clip\'s source is missing, or cannot be read as far as the clip uses it.',
'Read the project and check each clip\'s source: it must exist and play, and in + frames must stay '
'within its real length. Correct the project, then export the new revision.'),
@@ -239,7 +247,7 @@ def recovery(code, *, kind=None, stage='request', resumable=True):
return 'Tell the user to free disk space, then submit the picture again with a new ref.'
if kind == 'music' and code == 'out_of_memory':
return 'Close other GPU applications, then submit the music job again with a new ref.'
- if kind == 'stems' and code == 'out_of_memory':
+ if kind in ('stems', 'voice', 'voice_lines') and code == 'out_of_memory':
return 'Close other GPU applications, then submit the job again with a new ref.'
if kind == 'image' and code == 'no_memory':
return 'Lower the picture dimensions or use fewer reference pictures, then submit with a new ref.'
diff --git a/freevideo_engine/agent/guide.py b/freevideo_engine/agent/guide.py
index 81726d2..cd6ce81 100644
--- a/freevideo_engine/agent/guide.py
+++ b/freevideo_engine/agent/guide.py
@@ -145,6 +145,7 @@ GUIDE = {
'POST /projects/{id}/generate reserves pending parts for clip_ids or a start/frames range and fills them as the returned job finishes. POST /projects/{id}/export renders the revision you read; wait for its job.',
'Music: POST /projects/{id}/music with the revision you read and a caption; wait for the job, then read the project: result.clip_ids are the new clips on a2. Run it after the cut is joined; a second run needs the earlier score removed from a2.',
'Split sound: POST /projects/{id}/stems with the revision you read and the clip_id of a sound clip; wait for the job. The clip\'s sound is split into dialogue, music and ambience on tracks a4, a5 and a6 over the clip\'s range, and the clip itself goes silent. result.parts lists each track with its name, clip_id, start, end and frames. Offered only where features lists projects.stems; a clip is split once, and deleting the three parts gives its sound back. To save the three tracks and the original mix as four wav files, POST /projects/{id}/stems/export with the revision and the split clip\'s clip_id; the answer lists folder and files at once.',
+ 'Change a voice: POST /projects/{id}/voice/lines with the revision you read and a selection (clip_id, and start and end in seconds if only part) to write down the lines; read result.lines. Change the text of the lines to redo and POST /projects/{id}/voice/replace with the same selection and all lines (original_text unchanged); only the changed lines are spoken again, in the voice of the original, and the rest keep their sound. result.lines gives where each new line starts and ends; a line that cannot fit before the next one keeps its original voice with replaced false and warning too_long. POST /projects/{id}/voice/estimate with the lines answers at once with the seconds the changed lines take. Offered only where features lists projects.voice.',
'Subtitles: POST /projects/{id}/listen with mode transcribe (writes down what is said) or align (times the lines you give) and the revision you read; wait for the job and read result.lines, each with start and end frames. Then add them with a subtitles operation of cues {start, frames, text}; replace true swaps the whole track. With no speech model the job fails with no_speech_model: write the lines yourself.',
'The user sees one line under each picture clip: its note, or else the first line of its description. A description is generation text, often long and in English, so after you add, generate or change a clip, write a note (operation note) in the user\'s language saying what is in the picture: the subject first, then the setting, for example "Train through morning mist" (in Chinese "小火车穿过晨雾"). A short clip shows only the first few characters, about four in Chinese, and those must already make sense on their own. Do not write where the clip sits (opening, middle, ending, part 2): the order of the clips shows that. Do not write how far along it is: the clip already shows its progress. Keep it true when you change the clip again.',
'Reuse ref or Idempotency-Key only for the identical request. Source video/audio/thumbnail URLs require the same Agent token.',
@@ -380,6 +381,10 @@ CAPABILITIES_ZH = {
'剪辑拼好后先读取 revision,并删去该轨道上原有的配乐;首次使用会下载音乐模型(约 20 GB)',
('POST', '/projects/{id}/stems'): '把一个声音片段的声音分成人声、音乐和环境声,放在其下方的 a4、a5、a6 三条轨道上,原片段静音。'
'先读取 revision 与片段 ID,等待返回的任务完成,结果 parts 给出各轨道的名称、起止帧;仅在 features 含 projects.stems 时可用',
+ ('POST', '/projects/{id}/voice/lines'): '把选中声音片段里说的话逐句写下来,每句带起止秒数;先读取 revision 与片段 ID,等待返回的任务完成,结果 lines 给出各句;仅在 features 含 projects.voice 时可用',
+ ('POST', '/projects/{id}/voice/replace'): '把改过文字的句子用原说话人的声音重新说一遍,放回原句的位置,没改的句子保持原声;提交 voice/lines 给出的全部句子,只改 text;结果 lines 给出每句新声的起止,放不进下一句之前的句子保留原声',
+ ('POST', '/projects/{id}/voice/shorten'): '按目标秒数改短一句的文字;只返回文字,不保存也不配音',
+ ('POST', '/projects/{id}/voice/estimate'): '估算改过的句子配出来的秒数;立即返回',
('POST', '/projects/{id}/stems/export'): '把已拆开片段的人声、音乐、环境声和原混音保存为四个 wav 文件;先读取 revision 与被拆片段的 ID,返回立即给出文件夹和文件名',
('POST', '/images'): '像图片编辑页面一样打开图片;提交 source,会话会保留源图片副本和图层文档',
('GET', '/images/{id}'): '读取图片编辑会话的图层文档、选区和 revision;编辑前先读取 revision 和图层标识',
diff --git a/freevideo_engine/agent/jobs.py b/freevideo_engine/agent/jobs.py
index 53d5b34..906d04a 100644
--- a/freevideo_engine/agent/jobs.py
+++ b/freevideo_engine/agent/jobs.py
@@ -32,7 +32,7 @@ from ..system import windows
ACTIVE = ('waiting', 'queued', 'running', 'cancelling')
CAST_AGAIN = 'POST /freevideo/agent/v1/cast/%s/generate to make the reference pictures again.'
DONE = ('succeeded', 'failed', 'cancelled', 'interrupted')
-GPU_KINDS = ('vlm', 'enhance', 'image', 'music', 'stems')
+GPU_KINDS = ('vlm', 'enhance', 'image', 'music', 'stems', 'voice')
IMAGE_ERRORS = errors.IMAGE_ERRORS
IMAGE_WAIT_ERRORS = ('busy', 'busy_video')
IMAGE_WAIT_NOTICES = {
@@ -532,7 +532,7 @@ class Jobs:
def readable(row):
return (isinstance(row, dict) and specs.JOB_ID.fullmatch(str(row.get('id')))
- and row.get('kind') in ('generate', 'assemble', 'export', 'listen', 'image_edit') + GPU_KINDS
+ and row.get('kind') in ('generate', 'assemble', 'export', 'listen', 'voice_lines', 'image_edit') + GPU_KINDS
and row.get('status') in ACTIVE + DONE and isinstance(row.get('spec'), dict)
and type(row.get('created_at')) in (int, float) and math.isfinite(row['created_at'])
and (self._receipts is None or (type(row.get('_receipt_revision', 0)) is int
@@ -601,7 +601,7 @@ class Jobs:
records = list(latest.values())
for row in records:
if readable(row):
- if row.get('status') == 'running' and row['kind'] in ('assemble', 'export', 'listen', 'image_edit') + GPU_KINDS:
+ if row.get('status') == 'running' and row['kind'] in ('assemble', 'export', 'listen', 'voice_lines', 'image_edit') + GPU_KINDS:
# This process restarted while the work ran in it; nothing continues it.
self.finish(row, 'failed', error=dict(code='interrupted',
message='ComfyUI restarted while this job ran; submit it again'), save=False)
@@ -1488,6 +1488,12 @@ class Jobs:
elif row['kind'] == 'stems':
from .stems import run_stems
result = run_stems(self, row)
+ elif row['kind'] == 'voice_lines':
+ from .voice import run_voice_lines
+ result = run_voice_lines(self, row)
+ elif row['kind'] == 'voice':
+ from .voice import run_voice_replace
+ result = run_voice_replace(self, row)
else:
result = self.run_assemble(row)
return result
diff --git a/freevideo_engine/agent/projects.py b/freevideo_engine/agent/projects.py
index 619935c..803cd88 100644
--- a/freevideo_engine/agent/projects.py
+++ b/freevideo_engine/agent/projects.py
@@ -74,6 +74,11 @@ class ProjectError(JobError):
return value
+def _as_expected(project, clip_id, expect):
+ from .voice import signature
+ return signature(_clips(project).get(clip_id)) == expect
+
+
def _invalid(path, expected, received=None):
return spec.SpecError([schema.diagnostic(path, expected, received)])
@@ -460,11 +465,75 @@ class Projects:
self._remember(ref, digest, result, client)
return result
- def lay_stems(self, identity, clip_id, sources):
- """Put the three parts on sub-tracks under the clip and silence the clip; returns the save and the new clip IDs."""
+ def _voice_job(self, name, kind, identity, value, *, jobs, ref, client):
+ """Queue a voice job for the revision the caller read; the selected clip must be a sound clip with media."""
+ schema.validate(schema.project_schema(name), value)
+ with self.listen_lock:
+ digest, replay = self._receipt('projects:' + name, dict(id=identity, body=value), ref, client)
+ if replay is not None:
+ return replay
+ stored = self._load(identity)['project']
+ if stored['revision'] != value['revision']:
+ raise ProjectError('project_changed', project=stored)
+ selection = value['selection']
+ clip = _clips(stored).get(selection['clip_id'])
+ if clip is None or not clip.get('source'):
+ raise _invalid('/selection/clip_id', 'a sound clip with media', selection['clip_id'])
+ if 'end' in selection and selection['end'] <= selection.get('start', 0):
+ raise _invalid('/selection/end', 'a time after the start', selection['end'])
+ if kind == 'voice':
+ ids = [line['id'] for line in value['lines']]
+ if len(set(ids)) != len(ids):
+ raise _invalid('/lines', 'lines with different ids', ids[:3])
+ for i, line in enumerate(value['lines']):
+ if line['end'] <= line['start']:
+ raise _invalid('/lines/%d/end' % i, 'a time after the start', line['end'])
+ from . import voice
+ if not value.get('recognition'):
+ raise _invalid('/recognition', 'the recognition voice_lines gave for these lines; run voice_lines first')
+ wrong = voice.check(value['recognition'], identity, selection['clip_id'], value['lines'])
+ if wrong:
+ raise _invalid(*wrong)
+ reference = (value.get('voice') or {}).get('reference')
+ if reference and reference['clip_id'] not in _clips(stored):
+ raise _invalid('/voice/reference/clip_id', 'a clip of this project', reference['clip_id'])
+ job = jobs.submit(dict(kind=kind, editor_project=identity,
+ **{k: v for k, v in value.items() if k != 'ref'}), client=client)[0]
+ result = dict(job=jobs.view(job), jobs=[jobs.view(job)])
+ self._remember(ref, digest, result, client)
+ return result
+
+ def voice_lines(self, identity, value, *, jobs, ref=None, client=None):
+ """Queue the writing down of a selection's lines (what is said, and when)."""
+ return self._voice_job('voice_lines', 'voice_lines', identity, value, jobs=jobs, ref=ref, client=client)
+
+ def voice_replace(self, identity, value, *, jobs, ref=None, client=None):
+ """Queue the new voice for the lines the caller changed."""
+ return self._voice_job('voice_replace', 'voice', identity, value, jobs=jobs, ref=ref, client=client)
+
+ def voice_estimate(self, identity, value, *, ref=None, client=None):
+ """Seconds the changed lines take to make; asked at once, nothing is stored."""
+ from . import voice
+ schema.validate(schema.project_schema('voice_estimate'), value)
+ self._load(identity)
+ return voice.estimate(value['lines'], value.get('limit'))
+
+ def voice_shorten(self, identity, value, *, ref=None, client=None):
+ """Rewriting a line to a length needs a text model; none is wired in this version."""
+ schema.validate(schema.project_schema('voice_shorten'), value)
+ self._load(identity)
+ raise JobError('shorten_unavailable', 'No text model is available to shorten the line')
+
+ def lay_stems(self, identity, clip_id, sources, *, expect=None):
+ """Put the three parts on sub-tracks under the clip and silence the clip; returns the save and the new clip IDs.
+
+ `expect` (voice.signature of the clip) makes it a no-op that raises project_changed when the clip is not as it was."""
before, refused = {}, {}
def change(project):
before['ids'] = set(_clips(project))
+ if expect is not None and not _as_expected(project, clip_id, expect):
+ refused['stale'] = True
+ return dict(ok=True, project=project)
reply = editor_runtime.runtime().call('placeStems', project, [clip_id, {
name: dict(kind='generated', id=source.split(':', 1)[1]) for name, source in sources.items()}],
module='model', serialize=True)[0]
@@ -473,11 +542,36 @@ class Projects:
return dict(ok=True, project=project)
return dict(ok=True, project=reply['value'])
saved = self._update(identity, change)
+ if refused.get('stale'):
+ raise JobError('project_changed', 'The clip changed while the voice was made')
if refused:
code = STATE_CODES.get(refused.get('code'), 'stems_failed')
raise JobError(code, 'The separated sound cannot be laid under the clip (%s)' % refused.get('code'))
return saved, sorted(set(_clips(saved)) - before['ids'])
+ def replace_voice_source(self, identity, clip_id, source, *, expect=None):
+ """The dialogue clip takes the file with the replaced voice; its place and range stay. Returns the save.
+
+ `expect` (voice.signature of the clip) makes it a no-op that raises project_changed when the clip is not as it was."""
+ refused = {}
+ def change(project):
+ if expect is not None and not _as_expected(project, clip_id, expect):
+ refused['stale'] = True
+ return dict(ok=True, project=project)
+ reply = editor_runtime.runtime().call('replaceSource', project, [clip_id, dict(
+ kind='generated', id=source.split(':', 1)[1])], module='model', serialize=True)[0]
+ if not reply['ok']:
+ refused.update(reply['error'])
+ return dict(ok=True, project=project)
+ return dict(ok=True, project=reply['value'])
+ saved = self._update(identity, change)
+ if refused.get('stale'):
+ raise JobError('project_changed', 'The clip changed while the voice was made')
+ if refused:
+ code = STATE_CODES.get(refused.get('code'), 'voice_failed')
+ raise JobError(code, 'The replaced voice cannot be put on the sound track (%s)' % refused.get('code'))
+ return saved
+
def lay_music(self, identity, source):
"""Put a finished score on a2 from frame 0 of the project as it is now; returns the save and the new clip IDs."""
found = comfy_editor.sources(self.root, [source])[source]
diff --git a/freevideo_engine/agent/routes.py b/freevideo_engine/agent/routes.py
index 79c82b5..7ef65d9 100644
--- a/freevideo_engine/agent/routes.py
+++ b/freevideo_engine/agent/routes.py
@@ -172,13 +172,27 @@ def submit_batch(jobs, body, client):
return jobs.submit_request(body, client=client)[0]
+JOB_OPERATIONS = ('generate', 'export', 'listen', 'music', 'stems', 'voice_lines', 'voice_replace')
+
+
def stems_supported(machine):
"""Sound separation is offered on NVIDIA machines with enough video memory, never on a Mac."""
from ..stems.service import supported
return supported(machine.get('vram_gib'), uuid=machine.get('gpu_uuid')) and machine.get('device_backend', 'cuda') == 'cuda'
-def features(continuation, vlm, stems=False):
+def voice_ready(port):
+ """Whether the dubbing model that changes a voice is on this machine."""
+ service = getattr(port.server, '_freevideo_dubbing', None)
+ if service is None:
+ return False
+ try:
+ return bool(service.info(wait=False).get('ready'))
+ except (OSError, ValueError):
+ return False
+
+
+def features(continuation, vlm, stems=False, voice=True):
"""What an agent can use here (ready says whether the engine can run now). A missing name is not offered."""
from .editor_runtime import available
rules = available()
@@ -190,7 +204,7 @@ def features(continuation, vlm, stems=False):
'enhance': bool(vlm.get('ready')), 'drama.film': True, 'images.edit': rules,
'images.layers': rules, 'images.heal': rules, 'images.render': rules, 'projects.read': True,
**{'projects.' + name: rules for name in ('insert', 'split', 'move', 'trim', 'remove', 'export', 'generate', 'listen', 'music')},
- 'projects.stems': bool(rules and stems)}
+ 'projects.stems': bool(rules and stems), 'projects.voice': bool(rules and stems and voice)}
def whitebox_support(jobs):
@@ -228,7 +242,7 @@ def capabilities(port, jobs):
return dict(
api_version=API_VERSION, freevideo_version=release or __version__, build=__version__, ready=ready, problem=problem,
continuation=continuation,
- features=features(continuation, vlm, stems=stems_supported(machine)),
+ features=features(continuation, vlm, stems=stems_supported(machine), voice=voice_ready(port)),
whitebox=whitebox_support(jobs),
device=dict(backend=machine.get('device_backend', 'cuda') if machine else None,
vram_limit_gib=machine.get('vram_gib'), ram_limit_gib=machine.get('ram_gib')),
@@ -241,6 +255,8 @@ def capabilities(port, jobs):
'listen': 'Listen to the sound of an editing project for subtitle lines and times',
'music': 'Compose music for an editing project and lay it on its second sound track',
'stems': 'Split the sound of a clip into dialogue, music and ambience on sub-tracks',
+ 'voice_lines': 'Write down the spoken lines of a sound clip with their start and end',
+ 'voice': 'Say edited lines again in the original voice and put them back where the originals were',
'image_edit': 'Render a saved revision of an image session to a picture'},
limits=dict(width=[256, 4096, 32], height=[256, 4096, 32], seconds=[1.625, 60 if continuation else BLOCK_SECONDS],
seconds_tested=[4, 15],
@@ -2225,7 +2241,7 @@ def register():
raise JobError('idempotency_conflict', 'Conflicting keys')
args = [] if operation == 'create' else [identity]
extra = dict(client=request.headers.get('X-FreeVideo-Client', '')[:80] or None)
- if operation in ('generate', 'export', 'listen', 'music', 'stems'):
+ if operation in JOB_OPERATIONS:
extra['jobs'] = jobs
if operation == 'generate':
def capability(size):
@@ -2235,9 +2251,9 @@ def register():
return None
extra['capability'] = capability
result = await asyncio.to_thread(getattr(projects, operation), *args, value, ref=key or ref, **extra)
- if operation in ('generate', 'export', 'listen', 'music', 'stems'):
+ if operation in JOB_OPERATIONS:
await jobs.tick()
- return reply(result, 201 if operation in ('create', 'generate', 'export', 'listen', 'music', 'stems') else 200)
+ return reply(result, 201 if operation == 'create' or operation in JOB_OPERATIONS else 200)
return guarded(handler)
for operation, (method, suffix, _) in contract.PROJECT_OPERATIONS.items():
diff --git a/freevideo_engine/agent/schema.py b/freevideo_engine/agent/schema.py
index ecbef87..ff7bab3 100644
--- a/freevideo_engine/agent/schema.py
+++ b/freevideo_engine/agent/schema.py
@@ -351,6 +351,58 @@ def stems(*, local=False):
**{'x-job-kind': 'stems'})
+VOICE_SELECTION = obj(dict(clip_id=string(64), start=number(0, 36000, description='Seconds from the clip start; default 0.'),
+ end=number(0, 36000, description='Seconds from the clip start; default the clip end.')), ('clip_id',),
+ description='The sound clip, or part of it, whose voice to change: the original sound, or its dialogue sub-track.')
+VOICE_LANGUAGE = string(8, pattern=r'^(auto|[a-z]{2,3})$', default='auto')
+VOICE_LINE = obj(dict(id=string(16, pattern=r'^l[0-9]{3,5}$'), start=number(0, 36000), end=number(0, 36000),
+ text=string(2000, description='The line as the user wants it said.'),
+ original_text=string(2000, description='The line as it was recognised.'),
+ rate=number(.1, 60, description='Units spoken per second, from voice_lines.')),
+ ('id', 'start', 'end', 'text', 'original_text'))
+VOICE_REFERENCE = obj(dict(clip_id=string(64), start=number(0, 36000), end=number(0, 36000), text=string(1000)),
+ ('clip_id', 'start', 'text'), description='A stretch of a sound clip whose voice every line takes.')
+VOICE_LINES_FIELDS = dict(
+ selection=VOICE_SELECTION, language=VOICE_LANGUAGE,
+ separate_first={'type': 'boolean', 'default': False,
+ 'description': 'Separate the sound first and recognise on the dialogue alone.'})
+VOICE_FIELDS = dict(
+ selection=VOICE_SELECTION, language=VOICE_LANGUAGE,
+ lines=array(VOICE_LINE, 1, 200, description='The lines voice_lines gave, with the edited ones changed; only those are made again.'),
+ voice=obj(dict(preset=string(64), reference=VOICE_REFERENCE),
+ description='Omit for each line to keep its own speaker\'s voice.'),
+ recognition=string(64, description='The recognition voice_lines gave (required); the lines are checked against it. When it is no longer kept, run voice_lines again.'),
+ seed=number(0, 2**31 - 1, integer=True, description='Same seed and lines, same voice.'),
+ lay={'type': 'boolean', 'default': True,
+ 'description': 'False leaves the new voice unplaced: result.apply says how the page puts it on the timeline.'})
+VOICE_SHORTEN_FIELDS = dict(
+ line=obj(dict(id=string(16, pattern=r'^l[0-9]{3,5}$'), text=string(2000), seconds=number(.1, 600)), ('id', 'text', 'seconds')),
+ target_seconds=number(.1, 600), language=VOICE_LANGUAGE,
+ context=array(string(2000), 0, 6, description='The lines before and after, for sense.'))
+VOICE_ESTIMATE_FIELDS = dict(
+ lines=array(obj(dict(id=string(16, pattern=r'^l[0-9]{3,5}$'), text=string(2000), original_text=string(2000),
+ rate=number(.1, 60), start=number(0, 36000), end=number(0, 36000)),
+ ('id', 'text', 'original_text')), 1, 200,
+ description='Every line of the selection; with start and end on the lines, each changed line also gets the room it has.'),
+ limit=number(0, 36000, description='The end of the selection in seconds, where the last line may reach; with the lines\' start and end.'))
+
+
+def voice_lines(*, local=False):
+ return obj(dict(kind={'const': 'voice_lines', 'type': 'string'},
+ editor_project=string(34, pattern=r'^e-[0-9a-f]{32}$'), revision=number(1, 2**53 - 1, integer=True),
+ **VOICE_LINES_FIELDS, **common()), ('kind', 'editor_project', 'revision', 'selection'),
+ description='Write down the lines of a selection with the local speech model; nothing is saved.',
+ **{'x-job-kind': 'voice_lines'})
+
+
+def voice(*, local=False):
+ return obj(dict(kind={'const': 'voice', 'type': 'string'},
+ editor_project=string(34, pattern=r'^e-[0-9a-f]{32}$'), revision=number(1, 2**53 - 1, integer=True),
+ **VOICE_FIELDS, **common()), ('kind', 'editor_project', 'revision', 'selection', 'lines'),
+ description='Say the edited lines again in the voice of the original and put them where the originals were.',
+ **{'x-job-kind': 'voice'})
+
+
def image_edit(*, local=False):
return obj(dict(kind={'const': 'image_edit', 'type': 'string'},
image=dict(IMAGE, description='The image session to render (POST /images).'),
@@ -361,12 +413,12 @@ def image_edit(*, local=False):
KINDS = dict(generate=generate, assemble=assemble, vlm=vlm, enhance=enhance, image=image,
- export=export, listen=listen, music=music, stems=stems, image_edit=image_edit)
+ export=export, listen=listen, music=music, stems=stems, voice_lines=voice_lines, voice=voice, image_edit=image_edit)
def job_schema(*, local=False, refs=False, batch=False):
# Music scores a joined cut, so it is not a storyboard row.
- value = {'oneOf': [f(local=local) for kind, f in KINDS.items() if not (batch and kind in ('music', 'stems'))], 'x-discriminator': 'kind',
+ value = {'oneOf': [f(local=local) for kind, f in KINDS.items() if not (batch and kind in ('music', 'stems', 'voice_lines', 'voice'))], 'x-discriminator': 'kind',
'description': 'x-discriminator selects field diagnostics by kind; all accepted branches use standard oneOf.'}
return batch_references(value) if refs else value
@@ -483,7 +535,7 @@ def contracts(*, local=False):
if method == 'GET':
extra['query'] = shape
else:
- extra.update(body=shape, idempotent=True, **({'status': 201} if name in ('create', 'generate', 'export', 'listen', 'music', 'stems') else {}))
+ extra.update(body=shape, idempotent=True, **({'status': 201} if name in ('create', 'generate', 'export', 'listen', 'music', 'stems', 'voice_lines', 'voice_replace') else {}))
routes[method, suffix] = route(summary, **extra)
for name, (method, suffix, summary) in WHITEBOX_OPERATIONS.items():
extra = dict(path=obj({'id': WHITEBOX}, ('id',))) if '{id}' in suffix else {}
@@ -874,13 +926,19 @@ PROJECT_OPERATIONS = {
'listen': ('POST', '/projects/{id}/listen', 'Listen to the timeline sound for subtitles: mode align times lines you give, transcribe writes down what is said. First get the revision; wait for the returned job, then add the lines with the subtitles edit. Nothing is saved; with no speech model for the language the call fails with no_speech_model, so write the lines by hand.'),
'stems': ('POST', '/projects/{id}/stems', 'Split one clip\'s sound into dialogue, music and ambience on tracks a4-a6. First get the revision and clip ID; wait for the job, then read result.parts.'),
'stems_export': ('POST', '/projects/{id}/stems/export', 'Save the separated dialogue, music and ambience of a split clip, with its original mix, as four wav files in the saved sounds. First get the revision and the split clip\'s ID; the answer lists the files at once.'),
+ 'voice_lines': ('POST', '/projects/{id}/voice/lines', 'Write down the lines of a selected sound clip, each with its start and end. First get the revision and clip ID; wait for the job, then read result.lines. Nothing is saved.'),
+ 'voice_replace': ('POST', '/projects/{id}/voice/replace', 'Say the lines you changed again in the original voice and put them where the originals were; unchanged lines keep their sound. Send the lines voice_lines gave, with text changed; wait for the job, then read result.lines for where each new line starts and ends.'),
+ 'voice_shorten': ('POST', '/projects/{id}/voice/shorten', 'Shorten one line to fit a number of seconds. The answer is a text; nothing is saved or spoken.'),
+ 'voice_estimate': ('POST', '/projects/{id}/voice/estimate', 'Estimate the seconds the changed lines take to make. The answer comes at once.'),
'music': ('POST', '/projects/{id}/music', 'Compose music as long as the joined cut and lay it on sound track a2 at frame 0, under the video sound. First get the revision; a2 must be empty. Wait for the returned job; the first use downloads the model (20 GB).'),
}
PROJECT_TOOLS = dict(list='freevideo_projects', get='freevideo_project_read', create='freevideo_project_create',
write='freevideo_project_write', operations='freevideo_editor_operations',
generate='freevideo_project_generate', export='freevideo_export',
listen='freevideo_project_listen', music='freevideo_project_music',
- stems='freevideo_project_stems', stems_export='freevideo_project_stems_export')
+ stems='freevideo_project_stems', stems_export='freevideo_project_stems_export',
+ voice_lines='freevideo_project_voice_lines', voice_replace='freevideo_project_voice_replace',
+ voice_shorten='freevideo_project_voice_shorten', voice_estimate='freevideo_project_voice_estimate')
SESSION_OPERATIONS = {
'create': ('POST', '/sessions', 'Create a work conversation for messages and plans shared with the user.'),
@@ -1003,6 +1061,14 @@ def project_schema(name):
return obj(dict(revision=revision, **MUSIC_FIELDS, ref=REF), ('revision', 'caption'))
if name in ('stems', 'stems_export'):
return obj(dict(revision=revision, **STEMS_FIELDS, **(STEMS_LAY if name == 'stems' else {}), ref=REF), ('revision', 'clip_id'))
+ if name == 'voice_lines':
+ return obj(dict(revision=revision, **VOICE_LINES_FIELDS, ref=REF), ('revision', 'selection'))
+ if name == 'voice_replace':
+ return obj(dict(revision=revision, **VOICE_FIELDS, ref=REF), ('revision', 'selection', 'lines', 'recognition'))
+ if name == 'voice_shorten':
+ return obj(dict(revision=revision, **VOICE_SHORTEN_FIELDS, ref=REF), ('revision', 'line', 'target_seconds'))
+ if name == 'voice_estimate':
+ return obj(dict(revision=revision, **VOICE_ESTIMATE_FIELDS, ref=REF), ('lines',))
if name != 'operations':
return EMPTY
fields = {
diff --git a/freevideo_engine/agent/spec.py b/freevideo_engine/agent/spec.py
index f19581f..ef3200a 100644
--- a/freevideo_engine/agent/spec.py
+++ b/freevideo_engine/agent/spec.py
@@ -300,6 +300,14 @@ def normalize(value, *, port=None):
elif kind == 'stems':
result.update(editor_project=value['editor_project'], revision=value['revision'], clip_id=value['clip_id'],
lay=value.get('lay', True))
+ elif kind == 'voice_lines':
+ result.update(editor_project=value['editor_project'], revision=value['revision'], selection=dict(value['selection']),
+ language=value.get('language', 'auto'), separate_first=bool(value.get('separate_first', False)))
+ elif kind == 'voice':
+ result.update(editor_project=value['editor_project'], revision=value['revision'], selection=dict(value['selection']),
+ lines=[dict(line) for line in value['lines']], language=value.get('language', 'auto'),
+ voice=dict(value.get('voice') or {}), lay=value.get('lay', True),
+ **{k: value[k] for k in ('seed',) if k in value})
elif kind == 'image_edit':
result.update(image=value['image'], revision=value['revision'])
else:
diff --git a/freevideo_engine/agent/voice.py b/freevideo_engine/agent/voice.py
new file mode 100644
index 0000000..63b739c
--- /dev/null
+++ b/freevideo_engine/agent/voice.py
@@ -0,0 +1,721 @@
+"""Voice change jobs: the lines of a selection, and new voice for the lines the user edited.
+
+The lines come from the separated voice (where each one starts and ends) and the recogniser (what it says).
+A replacement keeps every unedited sample of the separated voice and puts each new line into the window of the
+original one; a line that cannot fit before the next original line keeps its original voice.
+"""
+import hashlib
+import json
+import re
+import secrets
+import shutil
+import threading
+import time
+from collections import OrderedDict
+from math import gcd
+from pathlib import Path
+
+import numpy as np
+
+from .jobs import JobError
+from .stems import FOLDER as STEMS_FOLDER, _busy, run_stems
+
+FOLDER = 'freevideo-voice'
+FPS = 24
+RATE = 44100
+GAP_BEFORE = .05 # quiet kept before the first sound of a line, at most (half the gap to the line before)
+GAP_AFTER = .1 # quiet kept after the last sound of a line, at most (half the gap to the next line)
+CLEARANCE = .1 # a new voice stops this long before the next original line starts
+MIN_WINDOW = .3
+MARGIN = .1 # extra file read around a line when its sound is measured
+PREEMPTIONS = 8
+BUSY_WAITS = 8
+SPEED = dict(zh=4.5, en=2.8) # units per second of a voice, when the line gave no rate of its own
+DUB_CODES = {'out_of_memory': 'out_of_memory', 'model_missing': 'voice_model_missing',
+ 'invalid_request': 'invalid_request', 'disk_space': 'storage_unavailable'}
+SELECTION_MESSAGE = 'The selection must be a sound clip played at normal speed, forwards'
+
+
+def units(text):
+ """Spoken units of a text: every Chinese, Japanese or Korean character, and every other word."""
+ text = re.sub(r'[ -〿＀-/]', ' ', text)
+ wide = re.findall(r'[぀-ヿ㐀-鿿가-힯]', text)
+ rest = re.sub(r'[぀-ヿ㐀-鿿가-힯]', ' ', text)
+ return len(wide) + len(re.findall(r"[^\W_]+(?:['’-][^\W_]+)*", rest))
+
+
+def edited(lines):
+ """The lines whose text differs from the original after trimming."""
+ return [line for line in lines if line['text'].strip() != line['original_text'].strip()]
+
+
+def _room(lines, limit):
+ """{id: seconds a new voice may take from the line's start}, for lines that came with their times.
+
+ The same rule as the placing: the new voice may use the quiet after the line up to CLEARANCE before the next line
+ (or the end of the selection); the line's recognised start and end stand in for the measured onset and offset."""
+ timed = sorted((line for line in lines if line.get('start') is not None and line.get('end') is not None),
+ key=lambda line: line['start'])
+ room = {}
+ for position, line in enumerate(timed):
+ start, end = line['start'], line['end']
+ if position + 1 < len(timed):
+ after = timed[position + 1]['start']
+ elif limit is not None:
+ after = limit + CLEARANCE
+ else:
+ after = end + 2 * GAP_AFTER
+ trail = min(GAP_AFTER, max(0., (after - end) / 2))
+ last = max(end + trail, start + MIN_WINDOW)
+ room[line['id']] = max(last, after - CLEARANCE) - start
+ return room
+
+
+def estimate(lines, limit=None):
+ """Seconds each edited line takes to say: its units over the rate of the original line (or a default).
+
+ A line that came with its times also gets `available` (seconds from its start to the latest end of its new voice) and
+ `overrun` (seconds by which the new voice, at the fastest tempo, still passes that end; 0 when it fits)."""
+ from ..dubbing import align
+ room = _room(lines, limit)
+ result = []
+ for line in edited(lines):
+ text = line['text'].strip()
+ rate = line.get('rate')
+ if not rate or rate <= 0:
+ rate = SPEED['zh'] if re.search(r'[぀-ヿ㐀-鿿가-힯]', text) else SPEED['en']
+ seconds = round(max(units(text), 1) / rate, 2)
+ available = room.get(line['id'])
+ result.append(dict(id=line['id'], seconds=seconds,
+ available=None if available is None else round(available, 2),
+ overrun=None if available is None else round(max(0., seconds / align.MAX_STRETCH - available), 2)))
+ return dict(seconds=round(sum(row['seconds'] for row in result), 2), lines=result)
+
+
+# ---------------------------------------------------------------------------------------------- what was recognised
+
+_RECOGNIZED = OrderedDict() # token -> dict(project, clip, lines={id: (start, end, text)}, at)
+_RECOGNIZED_LOCK = threading.Lock()
+RECOGNIZED_KEEP = 64
+RECOGNIZED_SECONDS = 3600
+LINE_TOLERANCE = .011 # the lines are sent back rounded to milliseconds
+
+
+def remember(spec, ctx, language, lines):
+ """Keep what voice_lines recognised under a token: a hash of the project, the sound it read and every line.
+
+ The ids are positions in this one recognition, so a new recognition (or any change of the sound) gives a new token."""
+ rows = {line['id']: (line['start'], line['end'], line['text']) for line in lines}
+ source = ctx['source']
+ text = json.dumps([spec['editor_project'], ctx['clip']['id'], source['kind'], str(source['id']), ctx['base'],
+ ctx['start'], ctx['end'], language, sorted(rows.items())], ensure_ascii=False)
+ token = hashlib.sha256(text.encode()).hexdigest()[:16]
+ with _RECOGNIZED_LOCK:
+ _RECOGNIZED[token] = dict(project=spec['editor_project'], clip=ctx['clip']['id'], lines=rows, at=time.monotonic())
+ _RECOGNIZED.move_to_end(token)
+ while len(_RECOGNIZED) > RECOGNIZED_KEEP:
+ _RECOGNIZED.popitem(last=False)
+ return token
+
+
+def check(token, project, clip_id, lines):
+ """None when the lines agree with the recognition behind the token, else (path, expected, received) of the first
+ disagreement: a token that is unknown or too old (the service restarted, or other recognitions pushed it out), an id the
+ recognition never had, a line of the recognition that is not sent, another start or end, or another original text. Edited text may be anything; punctuation and width changes count as edits."""
+ with _RECOGNIZED_LOCK:
+ found = _RECOGNIZED.get(token)
+ if found is not None and time.monotonic() - found['at'] > RECOGNIZED_SECONDS:
+ del _RECOGNIZED[token]
+ found = None
+ if found is None:
+ return '/recognition', 'a recognition that is still kept; run voice_lines again for this clip', token
+ if found['project'] != project or found['clip'] != clip_id:
+ return '/recognition', 'a recognition of this clip', token
+ for index, line in enumerate(lines):
+ known = found['lines'].get(line['id'])
+ if known is None:
+ return '/lines/%d/id' % index, 'a line id from the recognition', line['id']
+ if abs(line['start'] - known[0]) > LINE_TOLERANCE:
+ return '/lines/%d/start' % index, 'the recognised start %s' % known[0], line['start']
+ if abs(line['end'] - known[1]) > LINE_TOLERANCE:
+ return '/lines/%d/end' % index, 'the recognised end %s' % known[1], line['end']
+ if line['original_text'].strip() != known[2].strip():
+ return '/lines/%d/original_text' % index, 'the recognised text', None
+ sent = {line['id'] for line in lines}
+ left = [name for name in found['lines'] if name not in sent]
+ if left:
+ # a new voice that runs on past a changed line must not land on a line that was left out
+ return '/lines', 'every line of the recognition (%d lines); missing %s' % (len(found['lines']), ', '.join(left[:3])), len(lines)
+ return None
+
+
+def signature(clip):
+ """What a result placed on a clip depends on: where it is, which sound it plays and how. None when it is gone."""
+ if not clip:
+ return None
+ source = clip.get('source') or {}
+ return dict(start=clip['start'], frames=clip['frames'], at=clip.get('in', 0), source=[source.get('kind'), source.get('id')],
+ speed=clip.get('speed') or 1, reverse=bool(clip.get('reverse')))
+
+
+# ---------------------------------------------------------------------------------------------- the selection
+
+def _tracks(project):
+ return [track for track in project['tracks'] if track['kind'] == 'audio']
+
+
+def context(jobs, spec):
+ """Where the sound of a selection is: {clip, parent, dialogue, separated, base, start, end, source, wav}."""
+ from ..editor_export import resolve_source
+ try:
+ project = jobs.projects.get(spec['editor_project'])['project']
+ except FileNotFoundError as error:
+ raise JobError('not_found', 'The project does not exist') from error
+ found = {}
+ for track in _tracks(project):
+ for clip in track['clips']:
+ found[clip['id']] = (track, clip)
+ selection = spec['selection']
+ if selection['clip_id'] not in found:
+ raise JobError('invalid_request', 'The selection names a clip that is not in this project')
+ track, clip = found[selection['clip_id']]
+ if track.get('stem') in ('music', 'ambience'):
+ raise JobError('invalid_request', 'The music and ambience parts have no voice to change')
+ if not clip.get('source'):
+ raise JobError('invalid_request', 'The clip is not a sound clip with media')
+ if (clip.get('speed') or 1) != 1 or clip.get('reverse'):
+ raise JobError('invalid_request', SELECTION_MESSAGE)
+ if clip['source']['kind'] == 'generated' and str(clip['source']['id']).startswith('freevideo-dubbing/'):
+ raise JobError('invalid_request', 'A written line is dubbed again from its text, not by changing its voice')
+ parent, dialogue = clip, None
+ if track.get('stem') == 'dialogue':
+ dialogue = clip
+ if clip.get('stem_of') not in found:
+ raise JobError('invalid_request', 'The voice part has lost the clip it was split from')
+ parent = found[clip['stem_of']][1]
+ else:
+ for other, candidate in found.values():
+ if (other.get('stem') == 'dialogue' and candidate.get('stem_of') == clip['id']
+ and candidate['start'] <= clip['start'] and candidate['start'] + candidate['frames'] >= clip['start'] + clip['frames']):
+ dialogue = candidate
+ break
+ length = clip['frames'] / FPS
+ start = float(selection.get('start', 0.))
+ end = float(selection.get('end', length))
+ if not 0 <= start < end <= length + 1e-6:
+ raise JobError('invalid_request', 'The selection must lie inside the clip')
+ carrier = dialogue or clip
+ try:
+ found_source = resolve_source('%s:%s' % (carrier['source']['kind'], carrier['source']['id']))
+ except (OSError, ValueError) as error:
+ raise JobError('voice_failed', 'The clip media is not readable') from error
+ path = found_source['audio']
+ if path is None:
+ raise JobError('voice_failed', 'The clip has no sound')
+ wav = Path(path).with_suffix('.wav')
+ return dict(clip=clip, parent=parent, dialogue=dialogue, separated=dialogue is not None,
+ base=carrier.get('in', 0) / FPS, start=start, end=min(end, length),
+ source=carrier['source'], path=wav if wav.is_file() else Path(path))
+
+
+def original_of(ctx, port):
+ """The wav of the voice before any change (a second change takes its references from it), and its source."""
+ if not ctx['separated']:
+ return ctx['path'], None
+ note = ctx['path'].parent / 'voice.json'
+ if ctx['source']['kind'] == 'generated' and note.is_file():
+ try:
+ original = json.loads(note.read_text(encoding='utf-8'))['original']
+ from ..editor_export import resolve_source
+ path = Path(resolve_source(original)['audio'])
+ wav = path.with_suffix('.wav')
+ return (wav if wav.is_file() else path), original
+ except (OSError, ValueError, KeyError, TypeError):
+ pass
+ return ctx['path'], '%s:%s' % (ctx['source']['kind'], ctx['source']['id'])
+
+
+# ---------------------------------------------------------------------------------------------- helpers
+
+class _Row:
+ """The real job row, with the request spec swapped: lets the stem job run inside another job without laying."""
+ def __init__(self, row, spec):
+ self._row, self.spec = row, spec
+
+ def __getitem__(self, key):
+ return self.spec if key == 'spec' else self._row[key]
+
+ def __setitem__(self, key, value):
+ self._row[key] = value
+
+ def get(self, key, default=None):
+ return self.spec if key == 'spec' else self._row.get(key, default)
+
+
+def _cancelled(row):
+ if row.get('_cancel'):
+ raise JobError('cancelled', 'Cancelled')
+
+
+def _phase(row, name, **detail):
+ row['_phase'] = name
+ row['_detail'] = json.dumps(detail) if detail else None
+
+
+def _separate(jobs, row, ctx, poll=.5):
+ """Split the sound of the parent clip without laying the parts; returns the sources and the dialogue wav."""
+ from ..editor_export import resolve_source
+ spec = dict(row['spec'], clip_id=ctx['parent']['id'], lay=False)
+ _phase(row, 'separating')
+ result = run_stems(jobs, _Row(row, spec), poll=poll)
+ sources = result['sources']
+ mp4 = Path(resolve_source(sources['dialogue'])['audio'])
+ return sources, (mp4.with_suffix('.wav') if mp4.with_suffix('.wav').is_file() else mp4)
+
+
+def _drop_folder(jobs, sources):
+ if not sources:
+ return
+ first = next(iter(sources.values()))
+ relative = first.split(':', 1)[1].rsplit('/', 1)[0]
+ if relative.startswith(STEMS_FOLDER + '/') and relative.count('/') == 1:
+ shutil.rmtree(jobs.port.output_path(relative + '/x.mp4', create=False).parent, ignore_errors=True)
+
+
+def _read(path, first, last):
+ """Mono float32 samples at 44.1 kHz of [first, last) seconds of a file, and the rate."""
+ from ..stems.separate import read_audio
+ audio, _ = read_audio(path)
+ low, high = max(0, int(first * RATE)), min(audio.shape[1], int(last * RATE))
+ return audio[:, low:high].mean(axis=0)
+
+
+def _resample(samples, rate):
+ if rate == RATE:
+ return np.asarray(samples, dtype=np.float32)
+ from scipy.signal import resample_poly
+ divisor = gcd(RATE, int(rate))
+ return resample_poly(np.asarray(samples, dtype=np.float32), RATE // divisor, int(rate) // divisor).astype(np.float32)
+
+
+# ---------------------------------------------------------------------------------------------- voice_lines
+
+def run_voice_lines(jobs, row):
+ """The lines of a selection: where each one is (from the voice) and what it says (from the recogniser)."""
+ from .. import subtitle_ears
+ from .. import subtitle_sherpa # noqa: F401 registers the recogniser
+ from ..voice import sentences
+ spec = row['spec']
+ ctx = context(jobs, spec)
+ sources = None
+ path = ctx['path']
+ try:
+ if spec.get('separate_first') and not ctx['separated']:
+ sources, path = _separate(jobs, row, ctx)
+ _cancelled(row)
+ try:
+ engine = subtitle_ears.engine_for('transcribe')
+ except subtitle_ears.EarsError as error:
+ raise JobError('no_speech_model', 'No speech model is installed on this machine') from error
+ _phase(row, 'recognizing', fraction=0.)
+
+ def cancelled():
+ return bool(row.get('_cancel'))
+
+ def progress(fraction):
+ _phase(row, 'recognizing', fraction=round(float(fraction), 3))
+ try:
+ whole = subtitle_ears.decode_mono(path, cancelled=cancelled)
+ rate = subtitle_ears.RATE
+ low, high = int((ctx['base'] + ctx['start']) * rate), int((ctx['base'] + ctx['end']) * rate)
+ audio = np.ascontiguousarray(whole[low:high])
+ heard = engine.transcribe(audio, spec.get('language', 'auto'), progress, cancelled) if len(audio) > rate * .2 \
+ else dict(language=spec.get('language', 'auto'), lines=[])
+ except subtitle_ears.Cancelled as error:
+ raise JobError('cancelled', 'Cancelled') from error
+ except subtitle_ears.EarsError as error:
+ raise JobError(('invalid_request' if error.reason in ('no_sound', 'too_long') else 'voice_failed'),
+ 'The selection could not be recognised (%s)' % error.reason) from error
+ words = [word for line in heard['lines'] for word in line['words']]
+ rows = sentences.lines(words, audio, rate)
+ lines = []
+ for index, line in enumerate(rows, 1):
+ lines.append(dict(id='l%03d' % index, start=round(line['start'], 3), end=round(line['end'], 3), text=line['text'],
+ speaker=0, words=line['words'],
+ rate=round(max(len(line['words']), 1) / max(line['end'] - line['start'], .1), 2)))
+ kind = 'dialogue' if ctx['separated'] or sources else 'a1'
+ return dict(editor_project=spec['editor_project'], revision=spec['revision'], separated=ctx['separated'],
+ language=heard.get('language'), speech=bool(lines), lines=lines, source=kind,
+ recognition=remember(spec, ctx, spec.get('language', 'auto'), lines) if lines else None)
+ finally:
+ _drop_folder(jobs, sources)
+
+
+# ---------------------------------------------------------------------------------------------- voice_replace
+
+
+
+def _seed(spec, lines):
+ if spec.get('seed') is not None:
+ return spec['seed']
+ text = '|'.join([spec['editor_project']] + ['%s=%s' % (line['id'], line['text']) for line in lines])
+ return int.from_bytes(hashlib.sha256(text.encode()).digest()[:4], 'big') % (2 ** 30)
+
+
+class Pace:
+ """Remaining seconds of the job: the average of the lines done so far, or the estimate before the first."""
+ def __init__(self, lines):
+ self.guess = {row['id']: row['seconds'] for row in estimate(lines)['lines']}
+ self.order = [line['id'] for line in lines]
+ self.times, self.mark_time = [], time.monotonic()
+
+ def mark(self):
+ now = time.monotonic()
+ self.times.append(now - self.mark_time)
+ self.mark_time = now
+
+ def left(self, done):
+ if self.times:
+ return round(sum(self.times) / len(self.times) * max(0, len(self.order) - done), 1)
+ return round(sum(self.guess.get(key, 0.) for key in self.order[done:]), 1)
+
+
+def _round(jobs, row, items, reference, seed, pace, base, count, done, poll=.25):
+ """One dubbing job for `items` ([{key, text, seconds, onset, offset, reference?}]).
+
+ Returns ({key: line row with 'data' (mono float32, 44.1 kHz) when a file was made}, cancelled)."""
+ from ..dubbing import wavio
+ service = getattr(jobs.port.server, '_freevideo_dubbing', None)
+ if service is None:
+ raise JobError('voice_model_missing', 'The dubbing model is not available on this machine')
+ names = {'d%d' % index: item['key'] for index, item in enumerate(items)}
+ lines = [dict({k: v for k, v in item.items() if k != 'key'}, id=name) for name, item in zip(names, items)]
+ value = dict(lines=lines, reference=reference, seed=seed)
+ waits, began = 0, None
+ while True:
+ _cancelled(row)
+ try:
+ key = service.start('dub', value, trusted=True, keep=True)['job']
+ began = None
+ except ValueError as error:
+ if str(error) == 'busy':
+ began = began or time.monotonic()
+ _phase(row, 'waiting', waited_seconds=int(time.monotonic() - began))
+ time.sleep(poll * 4)
+ continue
+ raise JobError(DUB_CODES.get(str(error), 'voice_failed'), 'The voice could not be made (%s)' % error) from None
+ cancelled, seen = False, 0
+ while True:
+ if row.get('_cancel') and not cancelled:
+ service.cancel(key)
+ cancelled = True
+ try:
+ state = service.status(key)
+ except ValueError:
+ raise JobError('voice_failed', 'The dubbing service lost the job') from None
+ phase = state.get('phase')
+ made = state.get('lines') or []
+ while seen < len(made):
+ pace.mark()
+ seen += 1
+ if phase in ('complete', 'failed', 'cancelled'):
+ break
+ if not cancelled:
+ if phase in ('starting', 'preparing'):
+ _phase(row, 'preparing')
+ elif phase in ('loading', 'loaded'):
+ _phase(row, 'loading')
+ else:
+ index = min(count, base + seen + 1)
+ _phase(row, 'dubbing', index=index, count=count, remaining_seconds=pace.left(base + seen),
+ line=pace.order[index - 1], done=done + [names.get(item['id']) for item in made])
+ time.sleep(poll)
+ waited = time.monotonic()
+ while service.busy() and time.monotonic() - waited < 60:
+ time.sleep(.1)
+ folder = service.folder
+ try:
+ if phase == 'failed':
+ if state.get('error') == 'busy' and waits < BUSY_WAITS:
+ waits += 1
+ waited = began = time.monotonic()
+ while _busy(jobs) and time.monotonic() - waited < 600:
+ _cancelled(row)
+ _phase(row, 'waiting', waited_seconds=int(time.monotonic() - began))
+ time.sleep(poll * 4)
+ began = None
+ continue
+ error = state.get('error')
+ raise JobError(DUB_CODES.get(error, 'voice_failed'), 'The voice could not be made (%s)' % error)
+ rows = {}
+ root = Path(service.input_directory())
+ for item in made:
+ target = dict(item, key=names[item['id']])
+ if item.get('file') and not item.get('code'):
+ samples, rate = wavio.read(root / item['file'])
+ target['data'] = _resample(samples[:, 0], rate)
+ rows[target['key']] = target
+ return rows, cancelled
+ finally:
+ if folder:
+ shutil.rmtree(folder, ignore_errors=True)
+
+
+def _measure(audio, first, last):
+ """(onset, offset) in file seconds of the sound in [first, last) of a mono 44.1 kHz file, or None."""
+ from ..dubbing import align
+ low, high = max(0, int(first * RATE)), min(len(audio), int(last * RATE))
+ span = align.energy_span(audio[low:high], RATE)
+ if span is None:
+ return None
+ return (low + span[0]) / RATE, (low + span[1]) / RATE
+
+
+def _windows(every, todo, spans, base, limit):
+ """Where each changed line's new voice goes: the original line plus a little quiet each side, and how far it may reach."""
+ windows, failures = {}, {}
+ for position, line in enumerate(every):
+ if line not in todo:
+ continue
+ onset, offset = spans[line['id']]
+ before = spans[every[position - 1]['id']][1] if position else None
+ after = spans[every[position + 1]['id']][0] if position + 1 < len(every) else limit + CLEARANCE
+ lead = min(GAP_BEFORE, max(0., (onset - before) / 2)) if before is not None else GAP_BEFORE
+ trail = min(GAP_AFTER, max(0., (after - offset) / 2))
+ first, last = max(0., onset - lead), offset + trail
+ if offset - onset < MIN_WINDOW / 2:
+ failures[line['id']] = 'failed'
+ continue
+ # nothing of this window reaches the quiet kept before the next line (where that one's own window starts)
+ hard = after if position + 1 == len(every) else after - min(GAP_BEFORE, max(0., (after - offset) / 2))
+ last = min(max(last, first + MIN_WINDOW), hard)
+ windows[line['id']] = dict(first=first, last=last, onset=onset, offset=offset, hard=hard,
+ deadline=max(last, after - CLEARANCE))
+ return windows, failures
+
+
+def _reference(jobs, spec, voice, every, original_wav, base):
+ """The job's reference (one that is always there), and whether lines keep their own."""
+ from ..editor_export import resolve_source
+ ref = voice.get('reference')
+ if ref:
+ project = jobs.projects.get(spec['editor_project'])['project']
+ other = next((clip for track in _tracks(project) for clip in track['clips'] if clip['id'] == ref['clip_id']), None)
+ if other is None or not other.get('source'):
+ raise JobError('invalid_request', 'The reference clip is not in this project')
+ try:
+ path = resolve_source('%s:%s' % (other['source']['kind'], other['source']['id']))['audio']
+ except (OSError, ValueError):
+ path = None
+ if path is None:
+ raise JobError('invalid_request', 'The reference clip has no sound')
+ picked = dict(file=str(path), text=ref.get('text', ''), start=other.get('in', 0) / FPS + float(ref.get('start', 0.)))
+ if ref.get('end') is not None:
+ picked['end'] = other.get('in', 0) / FPS + float(ref['end'])
+ return picked, False
+ longest = max(every, key=lambda line: line['end'] - line['start'])
+ return dict(file=str(original_wav), text=longest['original_text'].strip(), start=base + longest['start'],
+ end=base + longest['end']), True
+
+
+def run_voice_replace(jobs, row, *, poll=.25):
+ from ..dubbing import align
+ from ..stems.separate import read_audio, write_mp4, write_wav
+ from ..voice import assemble
+ started = time.monotonic()
+ spec, port = row['spec'], jobs.port
+ voice = spec.get('voice') or {}
+ if voice.get('preset'):
+ raise JobError('invalid_request', 'Preset voices are not offered yet')
+ ctx = context(jobs, spec)
+ every = sorted(spec['lines'], key=lambda line: line['start'])
+ todo = edited(every)
+ if not todo:
+ raise JobError('invalid_request', 'No line was changed')
+ sources, separate_seconds, out = None, None, None
+ try:
+ separated_now = not ctx['separated']
+ expect = signature(ctx['parent'] if separated_now else ctx['dialogue'])
+ if separated_now:
+ began = time.monotonic()
+ sources, dialogue_wav = _separate(jobs, row, ctx)
+ separate_seconds = round(time.monotonic() - began, 1)
+ original_wav, original_source = dialogue_wav, sources['dialogue']
+ else:
+ dialogue_wav = ctx['path']
+ original_wav, original_source = original_of(ctx, port)
+ _cancelled(row)
+ _phase(row, 'preparing')
+ audio, total = read_audio(dialogue_wav)
+ mono = audio.mean(axis=0)
+ base = ctx['base']
+ spans = {}
+ for position, line in enumerate(every):
+ first, last = base + line['start'], base + line['end']
+ low, high = first - MARGIN, last + MARGIN
+ # a neighbour closer than the margin is not this line's sound: stop halfway between the two lines
+ if position and base + every[position - 1]['end'] > low:
+ low = min(first, (base + every[position - 1]['end'] + first) / 2)
+ if position + 1 < len(every) and base + every[position + 1]['start'] < high:
+ high = max(last, (last + base + every[position + 1]['start']) / 2)
+ spans[line['id']] = _measure(mono, low, high) or (first, last)
+ windows, failures = _windows(every, todo, spans, base, min(base + ctx['end'], total))
+ work = [line for line in todo if line['id'] in windows]
+ if not work:
+ raise JobError('voice_failed', 'None of the changed lines can be dubbed')
+ reference, own = _reference(jobs, spec, voice, every, original_wav, base)
+ seed = _seed(spec, todo)
+ pace = Pace(work)
+
+ def item(line, length, offset):
+ window = windows[line['id']]
+ made = dict(key=line['id'], text=line['text'].strip(), seconds=round(length, 3),
+ onset=round(window['onset'] - window['first'], 3), offset=round(offset - window['first'], 3))
+ if own and line['end'] - line['start'] >= 1. and line['original_text'].strip():
+ made['reference'] = dict(file=str(original_wav), text=line['original_text'].strip(),
+ start=base + line['start'], end=base + line['end'])
+ return made
+
+ first_round, cancelled = _round(jobs, row, [item(line, windows[line['id']]['last'] - windows[line['id']]['first'],
+ windows[line['id']]['offset']) for line in work],
+ reference, seed, pace, 0, len(work), [], poll)
+ # a line that is too long for its own window may reach into the quiet before the next original line
+ second_round, retry = {}, []
+ if not cancelled:
+ for line in work:
+ made, window = first_round.get(line['id']), windows[line['id']]
+ if made and 'data' in made and made.get('warning') == 'too_long' and window['deadline'] > window['last'] + .02:
+ shortest = window['offset'] - window['onset'] + float(made.get('overrun') or 0.)
+ room = window['deadline'] - window['onset']
+ if room >= shortest * .99:
+ natural = shortest * float(made.get('stretch') or 1.) * float(made.get('speed') or 1.)
+ retry.append((line, min(natural, room)))
+ if retry:
+ count = len(work) + len(retry)
+ pace.order = [line['id'] for line in work] + [line['id'] for line, _ in retry]
+ second_round, cancelled = _round(
+ jobs, row, [item(line, windows[line['id']]['deadline'] - windows[line['id']]['first'],
+ windows[line['id']]['onset'] + span) for line, span in retry],
+ reference, seed, pace, len(work), count, list(first_round), poll)
+ unfinished = {line['id'] for line, _ in retry} - set(second_round) if cancelled else set()
+ # what was made goes into the separated voice, and every changed line gets its measured result
+ _phase(row, 'placing')
+ edits, results = [], []
+ for line in every:
+ if line not in todo:
+ continue
+ result = dict(id=line['id'], text=line['text'].strip(), start=line['start'], end=line['end'],
+ replaced=False, overrun=0., fitted=False, warning=None)
+ results.append(result)
+ if line['id'] in failures:
+ result['code'] = failures[line['id']]
+ continue
+ window = windows[line['id']]
+ made = second_round.get(line['id']) or first_round.get(line['id'])
+ if made is None:
+ result['code'] = 'cancelled'
+ continue
+ result.update(available=round((window['deadline'] if line['id'] in second_round else window['last']) - window['onset'], 3),
+ speed=made.get('speed'), stretch=made.get('stretch'), gain_db=made.get('gain_db'),
+ overrun=round(float(made.get('overrun') or 0.), 3), warning=made.get('warning'))
+ if made.get('code') or 'data' not in made:
+ result['code'] = made.get('code') or 'failed'
+ continue
+ if made.get('overrun') or made.get('warning') == 'too_long':
+ result['warning'] = 'too_long'
+ continue
+ position = int(round(window['first'] * RATE))
+ data = made['data'][:max(0, int(window['hard'] * RATE) - position)]
+ span = align.energy_span(data, RATE)
+ if span is None:
+ result['code'] = 'failed'
+ continue
+ first, last = (position + span[0]) / RATE, (position + span[1]) / RATE
+ onset_error, offset_error = first - window['onset'], last - window['offset']
+ # the fades stay in the quiet: the original sound is before its onset and after its offset
+ fades = (min(assemble.CROSSFADE, max(0., window['onset'] - window['first'])),
+ min(assemble.CROSSFADE, max(0., (position + len(data)) / RATE - window['offset'])))
+ edits.append((position, data, fades))
+ result.update(replaced=True, onset=round(first - base, 3), offset=round(last - base, 3),
+ onset_error=round(onset_error, 3), offset_error=round(offset_error, 3),
+ fitted=abs(onset_error) <= align.FIT_SECONDS and abs(offset_error) <= align.FIT_SECONDS)
+ if not result['fitted'] and not result['warning'] and offset_error < -align.FIT_SECONDS:
+ # the line as placed ends before its slot does, whatever the dubbing measured before the cut
+ result['warning'] = 'too_short'
+ replaced = [item for item in results if item['replaced']]
+ if cancelled and not replaced:
+ raise JobError('cancelled', 'Cancelled')
+ if not replaced and not any(item['warning'] == 'too_long' for item in results):
+ code = next((item['code'] for item in results if item.get('code')), 'failed')
+ raise JobError(DUB_CODES.get(code, 'voice_failed'), 'No changed line could be dubbed (%s)' % code)
+ if not replaced:
+ # every changed line was too long: the voice would be the original again, so nothing is written or placed,
+ # and a separation made for this job goes too (shorten the lines, then change again)
+ _drop_folder(jobs, sources)
+ seconds = round(time.monotonic() - started, 1)
+ return dict(editor_project=spec['editor_project'], revision=spec['revision'],
+ clip_id=ctx['dialogue']['id'] if ctx['dialogue'] else None,
+ sources=dict(new=None, original=None if separated_now else original_source), separated_now=False, parts=[],
+ placed=False, reason='too_long', cancelled=bool(cancelled),
+ partial=any(item.get('code') == 'cancelled' for item in results) or bool(unfinished),
+ kept=len(every) - len(todo), failed=sum(1 for item in results if item.get('code') and item['code'] != 'cancelled'),
+ lines=results, seed=seed, apply=None, seconds=seconds, separate_seconds=separate_seconds,
+ dub_seconds=round(seconds - (separate_seconds or 0.), 1))
+ stitched = assemble.stitch(audio.T, edits, RATE).T
+ folder = '%s/%s-%s' % (FOLDER, row['id'], secrets.token_hex(2))
+ out = port.output_path(folder + '/dialogue.mp4', create=True).parent
+ write_wav(out / 'dialogue.wav', stitched, exact=True)
+ write_mp4(out / 'dialogue.mp4', stitched)
+ new_source = 'generated:%s/dialogue.mp4' % folder
+ (out / 'voice.json').write_text(json.dumps(dict(version=1, original=original_source, base=base, seed=seed,
+ lines=results), ensure_ascii=False), encoding='utf-8')
+ revision, parts = spec['revision'], []
+ clip_id = ctx['dialogue']['id'] if ctx['dialogue'] else None
+ laid = dict(sources, dialogue=new_source) if separated_now else None
+ apply = dict(op='placeStems', clip_id=ctx['parent']['id'], sources=laid) if separated_now \
+ else dict(op='replaceSource', clip_id=clip_id, source=new_source)
+ placed, reason = False, None
+ if spec.get('lay', True):
+ # the project may have moved on while the voice was made: it is placed only if the clip it goes on is as it was
+ try:
+ if separated_now:
+ saved, clip_ids = jobs.projects.lay_stems(spec['editor_project'], ctx['parent']['id'], laid, expect=expect)
+ fresh = set(clip_ids)
+ parts = [dict(track=track['id'], name=track['stem'], clip_id=clip['id'], start=clip['start'],
+ frames=clip['frames'], end=clip['start'] + clip['frames'])
+ for track in saved['tracks'] if track.get('stem') for clip in track['clips'] if clip['id'] in fresh]
+ clip_id = next((part['clip_id'] for part in parts if part['name'] == 'dialogue'), None)
+ else:
+ saved = jobs.projects.replace_voice_source(spec['editor_project'], clip_id, new_source, expect=expect)
+ revision, placed, apply = saved['revision'], True, None
+ except JobError as error:
+ if error.code != 'project_changed':
+ raise
+ # nothing was changed: the files stay, and the result says how to put them on the project as it is now
+ try:
+ revision = jobs.projects.get(spec['editor_project'])['project']['revision']
+ except FileNotFoundError as missing:
+ raise JobError('not_found', 'The project does not exist') from missing
+ reason = 'project_changed'
+ else:
+ # the page places it itself: the same check, read only, tells it whether the clip is still as it was
+ from .projects import _as_expected
+ try:
+ current = jobs.projects.get(spec['editor_project'])['project']
+ except FileNotFoundError as missing:
+ raise JobError('not_found', 'The project does not exist') from missing
+ if not _as_expected(current, ctx['parent']['id'] if separated_now else ctx['dialogue']['id'], expect):
+ revision, reason = current['revision'], 'project_changed'
+ seconds = round(time.monotonic() - started, 1)
+ return dict(editor_project=spec['editor_project'], revision=revision, clip_id=clip_id,
+ sources=dict(new=new_source, original=original_source), separated_now=separated_now, parts=parts,
+ placed=placed, reason=reason, cancelled=bool(cancelled),
+ partial=any(item.get('code') == 'cancelled' for item in results) or bool(unfinished), kept=len(every) - len(todo),
+ failed=sum(1 for item in results if item.get('code') and item['code'] != 'cancelled'),
+ lines=results, seed=seed, apply=apply, seconds=seconds, separate_seconds=separate_seconds,
+ dub_seconds=round(seconds - (separate_seconds or 0.), 1))
+ except BaseException:
+ if out is not None:
+ shutil.rmtree(out, ignore_errors=True)
+ _drop_folder(jobs, sources)
+ raise
diff --git a/freevideo_engine/stems/separate.py b/freevideo_engine/stems/separate.py
index 2059ab9..4a5d788 100644
--- a/freevideo_engine/stems/separate.py
+++ b/freevideo_engine/stems/separate.py
@@ -17,6 +17,11 @@ SAMPLE_RATE = 44100
CHUNK_SECONDS = 6.0
HOP_SECONDS = 3.0
BATCH = 4
+# The network leaves speech in the music part when the whole track is quiet (an old recording at -37 dBFS comes
+# back with an empty voice part). A quiet track is lifted to this level for the model; the parts are scaled back.
+QUIET_RMS = .04
+TARGET_RMS = .08
+MAX_LIFT = 12.
MIX_NAME = 'mix.wav'
# Model output name -> track name. The order is the order of the sub-tracks in the editor.
@@ -53,6 +58,14 @@ def model_function(model, device):
return run
+def lift_for_model(audio):
+ """The gain that brings a quiet track (float [channels, samples]) to the level the model separates well; 1 otherwise."""
+ level = float(np.sqrt(np.mean(np.square(audio, dtype=np.float64)))) if audio.size else 0.
+ if level <= 1e-5 or level >= QUIET_RMS:
+ return 1.
+ return min(TARGET_RMS / level, MAX_LIFT)
+
+
def separate_array(function, audio, *, chunk=None, hop=None, batch=BATCH, progress=lambda fraction: None):
"""audio: float tensor [channels, samples]; function maps chunks [n, 1, chunk] to {stem: [n, 1, chunk]}.
@@ -111,11 +124,12 @@ def read_audio(path):
return audio, audio.shape[1] / SAMPLE_RATE
-def write_wav(path, audio):
- """Atomically write float [2, samples] as 16-bit PCM at 44.1 kHz."""
+def write_wav(path, audio, *, exact=False):
+ """Atomically write float [2, samples] as 16-bit PCM at 44.1 kHz. `exact` scales by 32768, the divisor the readers use,
+ so samples read from a 16-bit file are written back unchanged (the default scales by 32767)."""
path = Path(path)
audio = np.asarray(audio)
- pcm = np.rint(np.clip(audio.T, -1, 1) * 32767).astype('<i2')
+ pcm = (np.clip(np.rint(audio.T * 32768), -32768, 32767) if exact else np.rint(np.clip(audio.T, -1, 1) * 32767)).astype('<i2')
temporary = path.with_name('.' + path.name + '.partial')
try:
with wave.open(str(temporary), 'wb') as output:
@@ -164,7 +178,10 @@ def separate_file(source, out_dir, weights, *, device='cuda', progress=lambda fr
model = load_model(weights, device)
if device == 'cuda':
torch.cuda.reset_peak_memory_stats()
- stems = separate_array(model_function(model, device), torch.from_numpy(audio), progress=progress)
+ lift = lift_for_model(audio)
+ stems = separate_array(model_function(model, device), torch.from_numpy(audio * np.float32(lift)), progress=progress)
+ if lift != 1.:
+ stems = {name: track / lift for name, track in stems.items()}
peak = torch.cuda.max_memory_allocated() / 2 ** 30 if device == 'cuda' else None
write_wav(out_dir / MIX_NAME, audio) # the original mix, exported next to the separated tracks
paths = {}
diff --git a/freevideo_engine/voice/__init__.py b/freevideo_engine/voice/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/freevideo_engine/voice/assemble.py b/freevideo_engine/voice/assemble.py
new file mode 100644
index 0000000..b04db63
--- /dev/null
+++ b/freevideo_engine/voice/assemble.py
@@ -0,0 +1,39 @@
+"""Put replacement lines into the original voice: the same length, the rest of the samples untouched."""
+import numpy as np
+
+CROSSFADE = .02
+
+
+def stitch(original, edits, rate, *, crossfade=CROSSFADE):
+ """`original`: float samples (frames, channels) or (frames,). `edits`: [(start_sample, samples)] or
+ [(start_sample, samples, (fade_in, fade_out))] where samples is mono or the same channels; each replaces
+ `original[start:start + len]` with a short crossfade at both edges. The fades are in seconds (`crossfade` when
+ not given); a caller with little quiet before or after a line gives a shorter fade so the fade stays in the quiet.
+
+ Everything outside the edits (and outside the fades) stays identical to the input; the length never changes."""
+ result = np.array(original, dtype=np.float32, copy=True)
+ flat = result.ndim == 1
+ if flat:
+ result = result[:, None]
+ for edit in edits:
+ start, voice = edit[0], edit[1]
+ fades = edit[2] if len(edit) > 2 else (crossfade, crossfade)
+ voice = np.asarray(voice, dtype=np.float32)
+ if voice.ndim == 1:
+ voice = np.repeat(voice[:, None], result.shape[1], axis=1)
+ elif voice.shape[1] != result.shape[1]:
+ voice = np.repeat(voice[:, :1], result.shape[1], axis=1)
+ start = max(0, int(start))
+ end = min(len(result), start + len(voice))
+ if end <= start:
+ continue
+ voice = voice[:end - start]
+ head = min(int(round(fades[0] * rate)), (end - start) // 2)
+ tail = min(int(round(fades[1] * rate)), (end - start) // 2)
+ ramp = np.ones(end - start, dtype=np.float32)
+ if head:
+ ramp[:head] = np.linspace(0., 1., head, endpoint=False, dtype=np.float32)
+ if tail:
+ ramp[-tail:] = np.linspace(0., 1., tail, endpoint=False, dtype=np.float32)[::-1]
+ result[start:end] = result[start:end] * (1. - ramp[:, None]) + voice * ramp[:, None]
+ return result[:, 0] if flat else result
diff --git a/freevideo_engine/voice/sentences.py b/freevideo_engine/voice/sentences.py
new file mode 100644
index 0000000..aaf64aa
--- /dev/null
+++ b/freevideo_engine/voice/sentences.py
@@ -0,0 +1,125 @@
+"""Sentences of a separated voice: where each line starts and ends, and what it says.
+
+The recogniser's word times are only a few tens of milliseconds sharp, and its word ends are capped, so they cannot
+mark a pause. The boundaries come from the separated voice itself: stretches of speech energy, joined across gaps
+shorter than `GAP`. The recogniser gives the words; a word belongs to the stretch that holds its middle.
+"""
+import numpy as np
+
+GAP = .25 # a silence at least this long ends a line (the pauses a replacement must keep)
+LONGEST = 29. # longest line, seconds (the longest reference the voice model takes)
+SHORTEST = .3 # a stretch shorter than this is a click or a breath, unless it holds a word
+FRAME = .01
+LEVEL = .06 # speech is above this share of the loud part of the voice (voiced RMS), ...
+FLOOR = 4. # ... and above this many times the quiet floor (10th percentile of the frames)
+PAD = .03 # soft onsets and the tail of the last sound sit just outside the loud frames
+
+
+def frame_levels(samples, rate):
+ """RMS of every 10 ms frame of a mono signal."""
+ size = max(1, int(round(rate * FRAME)))
+ count = len(samples) // size
+ if not count:
+ return np.zeros(0, dtype=np.float32)
+ return np.sqrt((np.asarray(samples[:count * size], dtype=np.float32).reshape(count, size) ** 2).mean(axis=1))
+
+
+def speech_runs(samples, rate, *, gap=GAP):
+ """[(start, end)] seconds of the voiced stretches; silences shorter than `gap` do not split a stretch."""
+ levels = frame_levels(samples, rate)
+ if not len(levels):
+ return []
+ loud = float(np.percentile(levels, 95))
+ if loud <= 1e-4:
+ return []
+ floor = float(np.percentile(levels, 10))
+ # a clip that is nearly all speech has a loud 10th percentile, so the floor term never passes a third of the loud part
+ threshold = max(LEVEL * loud, min(FLOOR * floor, .3 * loud))
+ voiced = levels > threshold
+ runs, begin = [], None
+ for index, flag in enumerate(voiced):
+ if flag and begin is None:
+ begin = index
+ elif not flag and begin is not None:
+ runs.append([begin, index])
+ begin = None
+ if begin is not None:
+ runs.append([begin, len(voiced)])
+ merged = []
+ for first, last in runs:
+ if merged and (first - merged[-1][1]) * FRAME < gap:
+ merged[-1][1] = last
+ else:
+ merged.append([first, last])
+ total = len(samples) / rate
+ return [(max(0., first * FRAME - PAD), min(total, last * FRAME + PAD)) for first, last in merged]
+
+
+def _join(words):
+ text = ''
+ for word in words:
+ piece = word['text']
+ if text and piece[:1].isascii() and piece[:1].isalnum() and text[-1].isascii():
+ text += ' '
+ text += piece
+ return text
+
+
+def _split_long(runs, samples, rate, longest):
+ """A stretch longer than `longest` is cut at its quietest 10 ms near the middle of what is left."""
+ levels = frame_levels(samples, rate)
+ result = []
+ for start, end in runs:
+ while end - start > longest:
+ low, high = int((start + longest * .5) / FRAME), int((start + longest) / FRAME)
+ window = levels[low:high]
+ cut = (low + int(np.argmin(window))) * FRAME if len(window) else start + longest
+ result.append((start, cut))
+ start = cut
+ result.append((start, end))
+ return result
+
+
+def lines(words, samples, rate, *, gap=GAP, longest=LONGEST):
+ """The lines of one voice: [{start, end, text, words}], in order, from word times and the voice samples.
+
+ `words` are [{text, start, end}]. A stretch without a word is not a line; a word outside every stretch
+ joins the nearest one, because the recogniser's times can lead or lag a little."""
+ runs = _split_long(speech_runs(samples, rate, gap=gap), samples, rate, longest)
+ held = [[] for _ in runs]
+ for word in words:
+ middle = (word['start'] + word['end']) / 2
+ if not runs:
+ break
+ best = min(range(len(runs)), key=lambda i: 0. if runs[i][0] <= middle <= runs[i][1]
+ else min(abs(middle - runs[i][0]), abs(middle - runs[i][1])))
+ held[best].append(word)
+ result = []
+ for (start, end), group in zip(runs, held):
+ if not group:
+ continue
+ group.sort(key=lambda word: word['start'])
+ result.append(dict(start=round(start, 3), end=round(end, 3), text=_join(group), words=group))
+ # a stretch too short to dub joins the neighbour it is closest to when that one is near enough
+ index = 0
+ while index < len(result):
+ row = result[index]
+ if row['end'] - row['start'] < SHORTEST and len(result) > 1:
+ before = result[index - 1] if index else None
+ after = result[index + 1] if index + 1 < len(result) else None
+ gaps = [(row['start'] - before['end'], index - 1) if before else None,
+ (after['start'] - row['end'], index + 1) if after else None]
+ gaps = [item for item in gaps if item]
+ near, neighbour = min(gaps)
+ if near < .6 and (row['end'] - result[neighbour]['start'] if neighbour < index
+ else result[neighbour]['end'] - row['start']) <= longest:
+ other = result[neighbour]
+ first, second = (other, row) if neighbour < index else (row, other)
+ merged = dict(start=first['start'], end=second['end'], text=_join(first['words'] + second['words']),
+ words=first['words'] + second['words'])
+ low = min(index, neighbour)
+ result[low:low + 2] = [merged]
+ index = max(0, low)
+ continue
+ index += 1
+ return result
diff --git a/web/editor.js b/web/editor.js
index fd7cfb1..d4aaa78 100644
--- a/web/editor.js
+++ b/web/editor.js
@@ -29,6 +29,8 @@ import {createSpeedComponent, readSpeedAudioSamples, handleSpeedKey, speedShortc
import {createEditorGenbar, loadEditorComponents, editorBlockStates, blockMessages} from './editor_genbar.js';
import {createEditorMusic} from './editor_music.js';
import {createEditorRecord, RECORD_ICON} from './editor_record.js';
+import {createEditorVoice} from './editor_voice.js';
+import {createVoiceStore, voiceAvailable} from './editor_voice_store.js';
import {MAX_MUSIC_TEXT as MUSIC_TEXT_LIMIT} from './editor_music_model.js';
import {createProgressConnection} from './progress_connection.js';
import {createSelectionActions, deletionStatus, isMultiple, isTyping, playheadIds, removableGaps, removeGaps} from './editor_select.js';
@@ -538,6 +540,40 @@ export function openEditor({api, app, from = null, panel = {}, components: given
}
// Music on the second sound track goes through the same bar; the bar shows a chosen piece's own job.
const music = createEditorMusic({state, store, t, completeSources, onStatus: () => genbar?.update(barValue)});
+ // Dubbing lines of a stretch of speech.
+ let voiceReady = false;
+ const generatedSource = value => { const match = /^generated:(.+)$/.exec(String(value || '')); return match ? {kind: 'generated', id: match[1]} : null; };
+ // Playing one line: the playhead goes to its start and the picture plays up to its end, then stops.
+ let lineStop = null;
+ function playVoiceLine(line) {
+ const found = voice.target(); if (!found) return;
+ lineStop?.();
+ const from = Math.round(found.clip.start + (found.start + line.start) * model.FPS), to = Math.round(found.clip.start + (found.start + line.end) * model.FPS);
+ state.seek(from); player.play();
+ const finish = () => { unwatch(); if (lineStop === finish) lineStop = null; };
+ const unwatch = state.subscribe(event => { if ((event.type ?? event) !== 'playhead') return; if (state.playhead >= to) { player.pause(); finish(); } else if (!player.playing) finish(); });
+ lineStop = finish;
+ }
+ const voice = createEditorVoice({state, store: createVoiceStore({api, state, queue}), t, onChange: () => genbar?.update(barValue), available: () => voiceReady, playLine: playVoiceLine, comparePlayback: async (clipId, name) => {
+ // Hearing the other take changes nothing in the edit: only the preview swaps the sound.
+ const source = generatedSource(name);
+ if (!source) { player.previewClipSource(clipId, null); return; }
+ await store.getSources([sourceKey(source)]);
+ player.previewClipSource(clipId, source);
+ }, place: async result => {
+ // What to put into the project is what the service says in `apply`; the plain shape is the same as replacing a source.
+ const apply = result.apply || {op: 'replaceSource', clip_id: result.clip_id, source: result.sources?.new};
+ const dubbed = generatedSource(result.sources?.new);
+ if (!dubbed || !apply.clip_id) throw new model.EditError('source');
+ if (apply.op === 'placeStems' && !model.placeStems) throw new model.EditError('source');
+ const stems = apply.op === 'placeStems' ? Object.fromEntries(model.STEM_TRACKS.map(name => [name, generatedSource(apply.sources?.[name])])) : null;
+ if (stems && (!model.placeStems || !Object.values(stems).every(Boolean))) throw new model.EditError('source');
+ const single = !stems && (generatedSource(apply.source) || dubbed);
+ await store.getSources([...(stems ? Object.values(stems) : [single])].map(sourceKey));
+ state.apply(t('Redub', '重新配音'), record => stems ? model.placeStems(record, apply.clip_id, stems) : model.replaceSource(record, apply.clip_id, single));
+ const found = stems && state.record.tracks.flatMap(track => track.clips).find(clip => clip.stem_of === apply.clip_id && clip.source?.id === stems.dialogue.id);
+ return {voiceClip: stems ? found?.id : apply.clip_id};
+ }});
let barValue = null;
const exportMatches = () => exportSnapshot && exportSnapshot.id === state.record.id && exportSnapshot.revision === state.record.revision && exportSnapshot.content === content(state.record);
function explainExport() {
@@ -1169,6 +1205,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given
async press(text, {allowEmpty = false} = {}) {
if (disposed || closing || initializing) return {say: t('Opening the edit…', '正在打开…')};
text = wellFormed(text);
+ if (voice.session() && ['ready', 'done'].includes(voice.session().phase)) { await voice.redub(); return {}; }
// A stretch of the music track, or a piece of music there: music, with or without words.
if (music.target()) {
if (characters(text) > MUSIC_TEXT_LIMIT) return {say: textLongText(t, MUSIC_TEXT_LIMIT), focus: true};
@@ -1252,6 +1289,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given
async cancel() {
if (music.target()) { await music.cancel(); return; }
if (stems?.active()) { await stems.cancel(); return; }
+ if (voice.session()?.phase === 'dubbing') { voice.cancel(); return; }
const range = selectionRange(state.selection);
const own = range && state.record.tracks[0].clips.find(clip => clip.job && generating(clip)
&& clip.start < range.start + range.frames && clip.start + clip.frames > range.start)?.job;
@@ -1380,6 +1418,8 @@ export function openEditor({api, app, from = null, panel = {}, components: given
return running;
}
+ voiceAvailable(api).then(value => { voiceReady = value; if (!disposed) voice.refresh(); });
+ actions.voice = {controller: voice};
actions.music = {target: selection => music.target(selection), preset: selection => music.preset(selection), choose: value => music.choose(value),
styles: music.styles, moods: music.moods, downloadBytes: music.downloadBytes};
genbar = createEditorGenbar(genHost, {state, t, actions, text: ''}); settleService();
@@ -1663,7 +1703,7 @@ export function openEditor({api, app, from = null, panel = {}, components: given
}
}
await store.activateProject(null).catch(() => {});
- rememberView(); disposed = true; music.dispose(); guide?.dispose(); transportControls.dispose(); toolbarMore.dispose(); player.dispose();
+ rememberView(); disposed = true; music.dispose(); voice.dispose(); guide?.dispose(); transportControls.dispose(); toolbarMore.dispose(); player.dispose();
textTools.dispose(); adjust.dispose(); notes.dispose(); pictureEdit.dispose();
audio.dispose(); stems?.dispose(); recording.dispose(); headerFit.dispose();
listeners.splice(0).forEach(remove => remove());
diff --git a/web/editor_audio.js b/web/editor_audio.js
index 305b2f6..576a384 100644
--- a/web/editor_audio.js
+++ b/web/editor_audio.js
@@ -342,12 +342,12 @@ export function createSpeechSpans(fetchSpans, {retry = 1000} = {}) {
}
// A picture's own sound: still joined to it, or taken from it and kept.
export const isOriginalSound = clip => !!clip.link || !!clip.detached_from;
-export function createAudioPreview({state, sources, api, volumePreview = new Map(), readSamples = null, error = () => {}, speech = null}) {
+export function createAudioPreview({state, sources, api, volumePreview = new Map(), sourcePreview = new Map(), readSamples = null, error = () => {}, speech = null}) {
let context = null, generation = 0, timer = null, identity = null, pending = false, disposed = false;
const cache = new Map(), nodes = new Set(); let origin = 0, cursor = 0, previousPosition = 0;
const ducking = (record, rate = 48000) => speech && sound.duckSources(record.tracks).length ? sound.duckRanges(record.tracks, speech.map, rate) : [];
const edited = record => record.tracks.some(track => track.muted || track.solo || track.clips.some(clip => clip.fade_in || clip.fade_out || clip.crossfade))
- || ducking(record).length > 0;
+ || sourcePreview.size > 0 || ducking(record).length > 0;
function stop() { generation++; clearTimeout(timer); timer = null; pending = false; identity = null; nodes.forEach(node => { try { node.stop(); } catch {} }); nodes.clear(); }
async function decoded(clip) {
const metadata = sources.get(sourceKey(clip.source)), url = mediaURL(api, metadata?.audio);
@@ -430,6 +430,7 @@ export function createAudioPreview({state, sources, api, volumePreview = new Map
render({record, position, playing, playbackRate = 1, silenceOriginal = false}) {
speech?.want(sound.duckSources(record.tracks));
if (!edited(record)) { if (identity) stop(); return false; }
+ if (sourcePreview.size) record = {...record, tracks: record.tracks.map(track => ({...track, clips: track.clips.map(clip => sourcePreview.has(clip.id) ? {...clip, source: sourcePreview.get(clip.id)} : clip)}))};
if (volumePreview.size) record = {...record, tracks: record.tracks.map(track => ({...track, clips: track.clips.map(clip => volumePreview.has(clip.id) ? {...clip, volume: volumePreview.get(clip.id)} : clip)}))};
// While a take is recorded the picture's own sound stays out of the mix.
if (silenceOriginal) record = {...record, tracks: record.tracks.map(track => ({...track, clips: track.clips.filter(clip => !isOriginalSound(clip))}))};
diff --git a/web/editor_genbar.js b/web/editor_genbar.js
index 2ba29e2..00e2c14 100644
--- a/web/editor_genbar.js
+++ b/web/editor_genbar.js
@@ -9,6 +9,8 @@ import {waitingText} from './chain_support.js';
import {showEmptyPrompt} from './empty_prompt_notice.js';
import {segmented} from './editor_adjust.js';
import {createPassMode} from './pass_mode.js';
+import {createVoicePanel} from './editor_voice.js';
+import * as dubbing from './editor_voice_model.js';
const el = (tag, cls, text) => { const e = document.createElement(tag); if (cls) e.className = cls; if (text != null) e.textContent = text; return e; };
export const timeEstimate = (seconds, t) => seconds == null ? '' : seconds < 60
? t(`about ${Math.max(1, Math.ceil(seconds))} s`, `约 ${Math.max(1, Math.ceil(seconds))} 秒`)
@@ -118,6 +120,9 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {
// Where a narrow English bar starts Generate on a line of its own (editor.css); hidden elsewhere.
const lineBreak = el('span', 'fv-ed-genbar-break'); lineBreak.setAttribute('aria-hidden', 'true');
generate.before(lineBreak); host.lang = t('en', 'zh-CN');
+ // Dubbing: the lines of a chosen stretch of speech take the description box's place.
+ const voice = actions.voice || null, voicePanel = voice ? createVoicePanel(host, voice.controller, {t}) : null;
+ if (voicePanel) description.before(voicePanel.element);
const failure = el('div', 'fv-ed-gen-failure fv-result-links'), failureBar = el('div', 'fv-ed-failure-actions fv-actions');
const failureReason = el('span', 'fv-ed-failure-reason'); failureReason.setAttribute('role', 'status');
const report = el('button', 'fv-ed-failure-report fv-action fv-action-secondary', t('Download diagnostic report', '下载诊断报告')); report.type = 'button';
@@ -176,6 +181,8 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {
report.hidden = !selectedFailure || selectedFailure.report === false;
const running = ['queued', 'running', 'cancelling'].includes(status?.state);
const musical = !!music?.target(selection);
+ const dub = !previewSelection && voice?.controller.target(selection) ? voice.controller.session(voice.controller.target(selection)) : null;
+ const voiced = !!dub && dub.phase !== 'none' && dub.phase !== 'error';
const selected = !!selection && (!!previewSelection || musical || (actions.supportedSelection?.() ?? true));
// While a part generates, choosing another place offers Generate again; it is queued behind.
// A press is being sent for the parts in `pressing`; only that same selection waits for it.
@@ -188,6 +195,7 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {
const opening = (starting || !!actions.selectionSending?.()) && !held;
const stopped = ['failed', 'cancelled', 'interrupted'].includes(status?.state);
host.dataset.mode = busy ? 'running' : selected ? 'selected' : 'idle';
+ host.dataset.voice = String(voiced); voicePanel?.update();
hint.hidden = !busy && !other;
// The description and Generate stay where they are while a part is made, greyed, so nothing around them moves;
// music being made shows only its progress and Cancel.
@@ -199,13 +207,17 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {
input.placeholder = !writable ? '' : musical ? t('Describe the music', '描述配乐') : sound ? t('Describe the picture', '描述画面') : t('Describe this part', '描述这一段');
if (sound) host.dataset.tourId = 'video-empty-range'; else delete host.dataset.tourId;
input.setAttribute('aria-label', t('Description', '描述'));
- generate.classList.toggle('fv-primary', selected && !busy);
+ generate.classList.toggle('fv-primary', (voiced ? voice.controller.changed(dub).length > 0 : selected) && !busy);
// A cancel already accepted is not offered again; the bar says it is stopping until the run ends.
const cancelling = status?.state === 'cancelling' && !held;
const finalization = clipFinalization(status);
cancel.hidden = !busy || opening || cancelling || !held && finalization?.cancelable === false; resume.hidden = !stopped;
const waiting = status?.state === 'queued' && !(status.blocks || []).some(block => ['queued', 'running'].includes(block.state) && block.prompt_id);
- if (opening) hint.textContent = t('Starting…', '正在开始…');
+ const dubbingNow = voiced && dub.phase === 'dubbing', finding = voiced && dub.phase === 'finding';
+ if (dubbingNow || finding) {
+ hint.hidden = false;
+ hint.textContent = finding ? t('Finding the lines', '正在识别台词') : dubbing.dubbingText(Math.max(1, dub.index), dub.count, dub.remaining, t);
+ } else if (opening) hint.textContent = t('Starting…', '正在开始…');
else if (held || waiting) hint.textContent = waiting && !held ? waitingText(status.waiting, t) : t('Queued', '排队中');
else if (cancelling) hint.textContent = t('Stopping…', '正在停止…');
else if (running) {
@@ -254,14 +266,17 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {
if (pass) passControl.sync(pass);
if (passMenu.hidden) passMenu.open = false;
}
+ if (voiced) generate.textContent = dubbing.redubLabel(voice.controller.changed(dub).length, t);
// Never disabled for a reason the press itself can explain or fix; only until the server says it can generate.
- generate.disabled = busy || !!previewSelection || !!actions.generationWaiting?.();
+ generate.disabled = busy || !!previewSelection || !!actions.generationWaiting?.() || (voiced && ((dub.phase !== 'ready' && dub.phase !== 'done') || !voice.controller.changed(dub).length));
+ // While it dubs only the progress sentence and Cancel stay; while it recognises there is nothing to press yet.
+ if (voiced) { generate.hidden = dub.phase === 'dubbing' || dub.phase === 'finding'; cancel.hidden = dub.phase !== 'dubbing'; cancel.disabled = false; }
resizeDescription();
input.dispatchEvent(new Event('change'));
generate.title = '';
if (assets) { if (selected && !busy && !musical) assets.refresh(); else assets.element.hidden = true; }
showMusicAssets(selected && !busy && musical && !previewSelection);
- if (selected && !busy) {
+ if (selected && !busy && !voiced) {
const key = JSON.stringify(selection);
if (estimateCache?.key === key) generate.textContent += estimateCache.label;
const seconds = await actions.estimate(selection);
@@ -370,7 +385,7 @@ export function createEditorGenbar(host, {state, t, actions, text = ''}) {
if (!value && note.contains(retryService)) { retryService.remove(); note.hidden = true; }
showService();
}, getText() { expand.commit(); return input.value; }, update(value) { status = value; render(); }, preview(selection) { previewSelection = selection; render(); }, setText(value) { input.value = value || ''; state.setRangeText(input.value); if (bound && input.value !== bound.text) { dirty = true; written = input.value; } render(); },
- dispose() { disposed = true; ++serial; clearTimeout(noteTimer); stopPass?.(); passControl?.dispose(); document.removeEventListener('pointerdown', closePass, true); clearTimeout(refNoticeTimer); duration.dispose(); unsubscribe(); resize?.disconnect(); expand.dispose(); references?.dispose(); assets?.dispose(); Object.values(pickers).forEach(picker => picker.dispose()); }};
+ dispose() { disposed = true; ++serial; clearTimeout(noteTimer); stopPass?.(); passControl?.dispose(); document.removeEventListener('pointerdown', closePass, true); clearTimeout(refNoticeTimer); duration.dispose(); unsubscribe(); resize?.disconnect(); expand.dispose(); references?.dispose(); voicePanel?.dispose(); assets?.dispose(); Object.values(pickers).forEach(picker => picker.dispose()); }};
}
// A missing optional module leaves the editor's existing progress in place.
diff --git a/web/editor_model.js b/web/editor_model.js
index 18216e1..76b2a2c 100644
--- a/web/editor_model.js
+++ b/web/editor_model.js
@@ -558,6 +558,14 @@ export function placeStems(record, id, sources) {
parent.stem_volume = volume; parent.volume = 0;
return finish(next);
}
+// A replaced voice: the part keeps its id, place and range and takes a file of the same length as its sound.
+export function replaceSource(record, id, source) {
+ const next = copy(record), clip = find(next, id);
+ check(clip && trackOf(next, id).kind === 'audio' && clip.source && !clip.pending, 'missing');
+ check(!trackOf(next, id).locked, 'locked');
+ clip.source = sourceData(source);
+ return finish(next);
+}
export function move(record, id, start, {targetTrack} = {}) {
if (isText(find(record, id))) return changeText(record, id, {start});
check(whole(start), 'range'); const selected = pair(record, id); check(selected.length, 'missing'); busy(selected);
diff --git a/web/editor_player.js b/web/editor_player.js
index b10d2bf..f5a546c 100644
--- a/web/editor_player.js
+++ b/web/editor_player.js
@@ -54,8 +54,8 @@ export function createEditorPlayer(host, transport, {state, sources, api, t, err
let seekGeneration = 0, forcePosition = false, pictureView = null, processedSound = false;
const chainSeconds = (chain, position) => Math.max(0, (position - chain.start) / 24 + (chain.offset || 0));
const playback = createPlaybackClock();
- const gains = new WeakMap(), volumePreview = new Map();
- const audioPreview = createAudioPreview({state, sources, api, volumePreview, readSamples: readAudioSamples, error, speech});
+ const gains = new WeakMap(), volumePreview = new Map(), sourcePreview = new Map();
+ const audioPreview = createAudioPreview({state, sources, api, volumePreview, sourcePreview, readSamples: readAudioSamples, error, speech});
const clipVolume = clip => volumePreview.get(clip?.id) ?? clip?.volume ?? 1;
function volume(media, value) {
// HTMLMediaElement stops at 1. A GainNode implements the record's 0–2 range.
@@ -457,6 +457,7 @@ export function createEditorPlayer(host, transport, {state, sources, api, t, err
setTextRenderer(callback) { textRenderer = callback; },
previewVolume(id, value) { volumePreview.set(id, value); render(); },
clearVolumePreview() { volumePreview.clear(); render(); },
+ previewClipSource(id, source) { if (source) sourcePreview.set(id, source); else sourcePreview.delete(id); render(); },
async previewSource(source) {
if (disposed) return;
const previousSize = previewSize || {width: state.record.width, height: state.record.height};
diff --git a/web/editor_voice.css b/web/editor_voice.css
new file mode 100644
index 0000000..e670c91
--- /dev/null
+++ b/web/editor_voice.css
@@ -0,0 +1,37 @@
+/* Dubbing lines in the generation bar. */
+.fv-ed-genbar[data-voice=true]{flex-wrap:wrap;align-items:center}
+.fv-ed-genbar[data-voice=true] .fv-ed-description-box,.fv-ed-genbar[data-voice=true] .fv-ed-first-frame{display:none}
+.fv-ed-genbar[data-voice=true] .fv-ed-gen-hint{flex:1}
+.fv-ed-genbar[data-voice=true] .fv-ed-generate{margin-left:auto}
+.fv-ed-voice{order:-1;flex:1 1 100%;min-width:0;display:flex;flex-direction:column;gap:8px}
+.fv-ed-voice[hidden]{display:none}
+.fv-ed-voice-top{display:flex;align-items:center;gap:12px;min-height:28px;font-size:13px;color:var(--fv-muted)}
+.fv-ed-voice-name{flex:1;min-width:0}
+.fv-ed-voice-compare{display:flex;border:1px solid var(--fv-border);border-radius:6px;overflow:hidden}
+.fv-ed-voice-compare[hidden]{display:none}
+.fv-editor .fv-ed-voice-compare button{height:26px;min-height:26px;padding:0 12px;border:0;border-radius:0;background:transparent;color:var(--fv-muted);font-size:12px}
+.fv-editor .fv-ed-voice-compare button[aria-pressed=true]{background:var(--fv-accent,#6db8fa);color:#0f1722}
+.fv-ed-voice-lines{display:flex;flex-direction:column;gap:6px;max-height:calc(6 * 44px);overflow-y:auto}
+.fv-ed-voice-line{display:flex;flex-direction:column;gap:2px}
+.fv-ed-voice-main{display:flex;align-items:flex-start;gap:8px;position:relative;padding-left:8px}
+.fv-ed-voice-mark{position:absolute;left:0;top:4px;bottom:4px;width:2px;border-radius:1px;background:transparent}
+.fv-ed-voice-line[data-changed=true] .fv-ed-voice-mark,.fv-ed-voice-line[data-active=true] .fv-ed-voice-mark{background:var(--fv-accent,#6db8fa)}
+.fv-ed-voice-line[data-active=true] .fv-ed-voice-mark{animation:fv-ed-voice-run 1.2s ease-in-out infinite}
+@keyframes fv-ed-voice-run{50%{opacity:.35}}
+@media (prefers-reduced-motion:reduce){.fv-ed-voice-line[data-active=true] .fv-ed-voice-mark{animation:none}}
+.fv-editor .fv-ed-voice-time{flex:none;width:44px;height:36px;min-height:36px;padding:0;border:0;background:transparent;color:var(--fv-muted);font-size:13px;font-variant-numeric:tabular-nums;font-family:ui-monospace,monospace;text-align:left}
+.fv-editor .fv-ed-voice textarea.fv-ed-voice-text{flex:1;min-width:0;resize:none;overflow:hidden;box-sizing:border-box;min-height:36px;max-height:none;line-height:20px;padding:8px;border:1px solid var(--fv-border);border-radius:6px;background:transparent;color:var(--fv-ink);font:inherit;font-size:13px;line-height:18px}
+.fv-ed-voice-extra{display:flex;align-items:center;gap:10px;padding-left:60px;font-size:12px;color:var(--fv-muted)}
+.fv-ed-voice-extra[hidden]{display:none}
+.fv-ed-voice-over{color:var(--fv-warn,#e0a84a)}
+.fv-ed-voice-old,.fv-ed-voice-short{color:var(--fv-muted)}
+.fv-ed-voice-line[data-failed=true] .fv-ed-voice-mark{background:var(--fv-warn,#e0a84a)}
+.fv-editor .fv-ed-voice-link{height:auto;min-height:0;padding:0;border:0;background:transparent;color:var(--fv-accent,#6db8fa);font-size:12px}
+.fv-ed-voice-placeholder{height:36px;border-radius:6px;background:var(--fv-border);opacity:.4}
+.fv-ed-voice-note{font-size:13px;color:var(--fv-muted)}
+.fv-ed-voice-note[hidden]{display:none}
+@media (max-width:640px){
+ .fv-editor .fv-ed-voice textarea.fv-ed-voice-text{font-size:16px}
+ .fv-ed-voice-extra{padding-left:60px;flex-wrap:wrap}
+}
+.fv-ed-voice-note:has(.fv-issue-button){display:flex;flex-wrap:wrap;align-items:center;gap:6px 8px}
diff --git a/web/editor_voice.js b/web/editor_voice.js
new file mode 100644
index 0000000..94319ec
--- /dev/null
+++ b/web/editor_voice.js
@@ -0,0 +1,216 @@
+// Dubbing lines through the generation bar: a stretch of speech chosen there shows its lines in place, each one
+// editable; only the lines changed are dubbed again. The server (or a stand-in) recognises and dubs; this keeps what
+// the user typed, the progress, the original/new choice for playback and the lines that did not fit.
+import * as voice from './editor_voice_model.js';
+import {isNetworkTypeError, connectionLostText} from './browser_extension.js';
+import {issueButton} from './issue_button.js';
+import {repairAction} from './repair_hint.js';
+
+const el = (tag, cls, text) => { const e = document.createElement(tag); if (cls) e.className = cls; if (text != null) e.textContent = text; return e; };
+
+// What a failed run says: the lines the user typed are kept, whatever went wrong; the code never reaches the words.
+export function failureCopy(error, t) {
+ if (isNetworkTypeError(error)) return {text: connectionLostText(t), report: false};
+ switch (error?.code) {
+ case 'out_of_memory': return {report: true, text: t('Not enough memory to finish dubbing. The edited lines are kept. Close other programs using the GPU, then redub.', '显存或内存不足,配音未完成,所改台词已保留。请关闭其他占用显卡的程序后重新配音。')};
+ case 'storage_unavailable': return {report: false, text: t('Could not write to disk, so dubbing did not finish. The edited lines are kept. Check the free disk space, then redub.', '无法写入磁盘,配音未完成,所改台词已保留。请检查磁盘空间后重新配音。')};
+ case 'voice_model_missing': return {report: false, text: t('The dubbing model is not fully installed. ', '配音模型没有装全。') + t(...repairAction())};
+ default: return {report: true, text: t('Dubbing did not finish. The edited lines are kept. Redub them to try again.', '配音未完成,所改台词已保留,请重新配音。')};
+ }
+}
+
+export function createEditorVoice({state, store, t, onChange = () => {}, place = async () => {}, available = () => true, playLine = () => {}, comparePlayback = () => {}}) {
+ const sessions = new Map();
+ let disposed = false, placing = false;
+ const target = (selection = state.selection) => available() && state.record && voice.voiceTarget(state.record, selection);
+ const keyOf = found => `${found.clipId}:${found.start}:${found.end}`;
+ const session = (found = target()) => found ? sessions.get(keyOf(found)) || null : null;
+ const emit = () => { if (!disposed) onChange(); };
+
+ // Recognition starts by itself when a stretch of speech is chosen.
+ async function open(found = target()) {
+ if (!found) return null;
+ const key = keyOf(found);
+ let current = sessions.get(key);
+ if (current) return current;
+ current = {key, found, phase: 'finding', lines: [], texts: new Map(), results: new Map(), shortened: new Map(), compare: 'new',
+ active: null, index: 0, count: 0, remaining: null, failed: 0, revision: state.record.revision};
+ sessions.set(key, current); emit();
+ try {
+ const result = await store.lines({found, revision: current.revision});
+ if (disposed || sessions.get(key) !== current) return current;
+ current.phase = result.speech && result.lines.length ? 'ready' : 'none';
+ current.lines = result.lines; current.separated = result.separated; current.recognition = result.recognition;
+ current.rate = new Map(result.lines.map(line => [line.id, line.rate]));
+ } catch (value) {
+ // A failed recognition is not kept: choosing the stretch again asks again.
+ current.phase = 'error'; current.error = value;
+ if (sessions.get(key) === current) sessions.delete(key);
+ if (!disposed) onChange(); return current;
+ }
+ emit(); return current;
+ }
+ const failedCount = current => [...current.results.values()].filter(value => value.state === 'failed').length;
+ const changed = current => voice.changedLines(current.lines, current.texts);
+ function edit(id, text, current = session()) {
+ if (!current || current.phase === 'dubbing') return;
+ // What the last run said about this line no longer fits its new words.
+ current.texts.set(id, text); current.results.delete(id); current.failed = failedCount(current); current.failure = null; emit();
+ }
+ async function redub(current = session()) {
+ if (!current || current.phase === 'dubbing' || current.phase === 'finding') return null;
+ const lines = changed(current);
+ current.phase = 'dubbing'; current.index = 0; current.count = lines.length; current.remaining = null; current.failed = 0; current.results = new Map(); current.notice = null; current.failure = null; emit();
+ try {
+ const send = () => store.replace({found: current.found, revision: current.revision, recognition: current.recognition,
+ lines: current.lines.map(line => ({id: line.id, start: line.start, end: line.end, text: current.texts.get(line.id) ?? line.text, original_text: line.text})),
+ progress: info => { current.index = info.index; current.count = info.count || current.count; current.active = info.line || null; current.remaining = info.remaining_seconds ?? current.remaining; emit(); }});
+ let result;
+ try { result = await send(); } catch (value) {
+ // The service no longer keeps the recognition (restart, age): recognise the same stretch again and put the typed words back by line id.
+ if (value?.code !== 'invalid_request') throw value;
+ const again = await store.lines({found: current.found, revision: current.revision});
+ const ids = new Set(again.lines.map(line => line.id));
+ if (again.lines.length !== current.lines.length || current.lines.some(line => !ids.has(line.id))) throw value;
+ current.lines = again.lines; current.recognition = again.recognition;
+ result = await send();
+ }
+ if (disposed) return result;
+ // The service did not put anything into the project: every line was too long, or the stretch was edited meanwhile.
+ if (result?.placed === false && (result.reason || !result.apply)) {
+ current.results = voice.lineResults(result); current.failed = result.failed || 0; current.phase = 'ready'; current.active = null; current.sources = null;
+ current.notice = result.reason === 'project_changed' ? 'changed' : null; current.revision = state.record.revision;
+ emit(); return result;
+ }
+ current.notice = null; current.failure = null;
+ placing = true;
+ try { current.voiceClip = (await place(result))?.voiceClip || current.found.clipId; } finally { placing = false; }
+ current.results = voice.lineResults(result); current.failed = result.failed || 0; current.sources = result.sources;
+ for (const line of current.lines) if (current.results.get(line.id)?.state === 'placed') line.text = current.texts.get(line.id) ?? line.text;
+ current.compare = 'new'; current.active = null; current.phase = 'done'; current.revision = state.record.revision;
+ emit(); return result;
+ } catch (value) {
+ current.phase = 'ready'; current.active = null; current.error = value;
+ // Said once, in the panel beside the lines: no pop-up, no second copy in the bar.
+ current.failure = value?.code === 'cancelled' ? null : value; emit();
+ return null;
+ }
+ }
+ function cancel(current = session()) { if (current?.phase === 'dubbing') { current.failure = null; store.cancel?.(); } }
+ function setCompare(value, current = session()) { if (!current?.sources) return; current.compare = value; comparePlayback(current.voiceClip || current.found.clipId, value === 'original' ? current.sources.original : null); emit(); }
+ // The model's rewrite of a line that did not fit; the original wording stays beside it until the user lets go of it.
+ async function shorten(id, current = session()) {
+ const line = current?.lines.find(value => value.id === id), result = current?.results.get(id);
+ if (!line || current.phase === 'dubbing' || current.shortened.get(id)?.busy) return;
+ const entry = {busy: true, original: current.texts.get(id) ?? line.text}; current.shortened.set(id, entry); current.failure = null; emit();
+ try {
+ const out = await store.shorten({line: {id, text: entry.original, seconds: result?.seconds}, target_seconds: Math.max(.1, (line.end - line.start))});
+ entry.busy = false; current.texts.set(id, out.text);
+ } catch (value) { current.shortened.delete(id); if (value?.code === 'shorten_unavailable') current.noShorten = true; }
+ emit();
+ }
+ function restore(id, current = session()) {
+ const entry = current?.shortened.get(id); if (!entry || entry.busy || current.phase === 'dubbing') return;
+ current.texts.set(id, entry.original); current.shortened.delete(id); current.failure = null; emit();
+ }
+ // Undo takes the new sound out again; the lines then are the old ones.
+ const stillPlaced = current => {
+ const clip = state.record.tracks.flatMap(track => track.clips).find(value => value.id === (current.voiceClip || current.found.clipId));
+ return !!clip && `generated:${clip.source?.id}` === String(current.sources?.new);
+ };
+ // The original is only heard while its stretch stays chosen; a changed project (undo included) makes the lines stale.
+ function clearCompare(current) { if (current.compare === 'original') { current.compare = 'new'; comparePlayback(current.voiceClip || current.found.clipId, null); } }
+ const unsubscribe = state.subscribe(event => {
+ const type = event.type ?? event;
+ const here = target() ? keyOf(target()) : null;
+ for (const [key, current] of [...sessions]) {
+ if (key !== here) clearCompare(current);
+ if (type === 'record' && !placing && current.phase === 'done' && !stillPlaced(current)) { clearCompare(current); sessions.delete(key); }
+ }
+ if (type === 'selection' || type === 'record') { const found = target(); if (found) open(found); emit(); }
+ });
+ // The entry appears once the engine says it can dub; a stretch already chosen is looked at then.
+ function refresh() { const found = target(); if (found) open(found); emit(); }
+ return {refresh, target, session, open, edit, redub, cancel, setCompare, shorten, restore, playLine,
+ dispose() { for (const current of sessions.values()) clearCompare(current); disposed = true; unsubscribe(); sessions.clear(); }, changed};
+}
+
+// The lines in the bar: a row each, time on the left, the words themselves to edit.
+export function createVoicePanel(host, controller, {t}) {
+ const link = el('link'); link.rel = 'stylesheet'; link.href = new URL('./editor_voice.css', import.meta.url).href; document.head.append(link);
+ const element = el('div', 'fv-ed-voice'), top = el('div', 'fv-ed-voice-top'), voiceName = el('span', 'fv-ed-voice-name', t('Lines', '台词'));
+ const compare = el('div', 'fv-ed-voice-compare'); compare.setAttribute('role', 'group');
+ const choices = [['original', t('Original', '原声')], ['new', t('New', '新声')]].map(([value, label]) => {
+ const button = el('button', null, label); button.type = 'button'; button.dataset.value = value;
+ button.onclick = () => controller.setCompare(value); compare.append(button); return button;
+ });
+ const list = el('div', 'fv-ed-voice-lines'), note = el('div', 'fv-ed-voice-note'); note.setAttribute('role', 'status');
+ top.append(voiceName, compare); element.append(top, list, note); element.hidden = true; host.append(element);
+ const rows = new Map();
+ function row(line) {
+ let entry = rows.get(line.id); if (entry) return entry;
+ const wrap = el('div', 'fv-ed-voice-line'), main = el('div', 'fv-ed-voice-main'), time = el('button', 'fv-ed-voice-time', voice.clockText(line.start)),
+ text = el('textarea', 'fv-ed-voice-text'), mark = el('span', 'fv-ed-voice-mark'), extra = el('div', 'fv-ed-voice-extra');
+ time.type = 'button'; time.setAttribute('aria-label', t('Play this line', '播放这一句'));
+ text.rows = 1; text.value = line.text; text.setAttribute('aria-label', t('Line', '台词'));
+ time.onclick = () => controller.playLine(line); text.oninput = () => { grow(text); controller.edit(line.id, text.value); };
+ text.onkeydown = event => { if (event.key === 'Enter' && !event.shiftKey && !event.isComposing) event.preventDefault(); };
+ main.append(mark, time, text); wrap.append(main, extra); entry = {wrap, text, time, mark, extra}; rows.set(line.id, entry); return entry;
+ }
+ // The box is border-box: the border is added to what the words need. Measured again when the bar's width or its style sheet changes.
+ const grow = area => { area.style.height = '0px'; area.style.height = (area.scrollHeight + area.offsetHeight - area.clientHeight) + 'px'; };
+ const regrow = () => { if (!element.hidden) list.querySelectorAll('textarea').forEach(grow); };
+ let seen = 0;
+ const observer = typeof ResizeObserver === 'function' ? new ResizeObserver(() => { if (list.clientWidth !== seen) { seen = list.clientWidth; regrow(); } }) : null;
+ observer?.observe(list);
+ link.onload = regrow;
+ function update() {
+ const found = controller.target(), current = found && controller.session(found);
+ element.hidden = !found || !current;
+ if (current && ['ready', 'done', 'dubbing'].includes(current.phase)) list.dataset.tourId = 'voice-lines'; else delete list.dataset.tourId;
+ if (element.hidden) return;
+ const dubbing = current.phase === 'dubbing', ready = ['ready', 'done', 'dubbing'].includes(current.phase);
+ compare.hidden = !current.sources;
+ for (const button of choices) button.setAttribute('aria-pressed', String(button.dataset.value === current.compare));
+ voiceName.hidden = !ready;
+ const failure = current.failure && failureCopy(current.failure, t);
+ note.replaceChildren();
+ if (failure) {
+ note.append(failure.text);
+ if (failure.report) note.append(issueButton(t, () => ({where: ['Editor generation', '剪辑器生成'], message: failure.text, kind: current.failure?.code || null})));
+ } else note.textContent = current.notice === 'changed' ? t('This part changed during dubbing, so the new voice was not placed. Redub it.', '这一段在配音期间有改动,新声未放入。请重新配音。') : current.phase === 'none' ? t('No lines were found in this part.', '这一段没有识别到台词。') : current.phase === 'done' && current.failed ? voice.failedText(current.failed, t) : '';
+ note.hidden = !note.textContent;
+ if (current.phase === 'finding') {
+ list.replaceChildren(...[0, 1, 2].map(() => el('div', 'fv-ed-voice-placeholder'))); rows.clear(); return;
+ }
+ if (!ready) { list.replaceChildren(); rows.clear(); return; }
+ const changedIds = new Set(controller.changed(current).map(line => line.id));
+ list.replaceChildren(...current.lines.map(line => {
+ const entry = row(line), result = current.results.get(line.id), short = current.shortened.get(line.id);
+ const value = current.texts.get(line.id) ?? line.text;
+ if (entry.text.value !== value) { entry.text.value = value; }
+ entry.text.disabled = dubbing; entry.text.readOnly = dubbing;
+ entry.wrap.dataset.changed = String(changedIds.has(line.id)); entry.wrap.dataset.active = String(current.active === line.id); entry.wrap.dataset.failed = String(result?.state === 'failed');
+ const extra = [];
+ if (short?.busy) extra.push(el('span', 'fv-ed-voice-shortening', t('Shortening', '正在改短')));
+ else if (short) {
+ const old = el('s', 'fv-ed-voice-old', short.original), undo = el('button', 'fv-ed-voice-link', t('Restore line', '恢复原句')); undo.type = 'button';
+ undo.onclick = () => controller.restore(line.id); extra.push(old, undo);
+ } else if (result?.state === 'kept' && result.overrun > 0) {
+ const over = el('span', 'fv-ed-voice-over', voice.overrunText(result.overrun, t, !current.noShorten));
+ extra.push(over);
+ if (!current.noShorten) {
+ const cut = el('button', 'fv-ed-voice-link', t('Shorten this line', '改短这句')); cut.type = 'button';
+ cut.onclick = () => controller.shorten(line.id); extra.push(cut);
+ }
+ } else if (result?.state === 'placed' && result.short) extra.push(el('span', 'fv-ed-voice-short', t('Shorter than the original, with a pause after it', '比原句短,句尾留有空白')));
+ entry.extra.replaceChildren(...extra); entry.extra.hidden = !extra.length;
+ return entry.wrap;
+ }));
+ regrow();
+ // A result that kept the original voice scrolls to its first line that did not fit; the list may hide it.
+ const over = current.lines.find(line => current.results.get(line.id)?.state === 'kept' && current.results.get(line.id).overrun > 0);
+ if (current.scrollOver !== current.results && over) { current.scrollOver = current.results; rows.get(over.id)?.wrap.scrollIntoView({block: 'nearest'}); }
+ }
+ return {element, update, dispose() { observer?.disconnect(); element.remove(); link.remove(); }};
+}
diff --git a/web/editor_voice_model.js b/web/editor_voice_model.js
new file mode 100644
index 0000000..7211135
--- /dev/null
+++ b/web/editor_voice_model.js
@@ -0,0 +1,71 @@
+// Dubbing lines of a chosen stretch of speech: which lines the user changed, what the bar says about them and which
+// ones did not fit. Pure helpers; the server does the recognition and the dubbing.
+import {FPS} from './editor_model.js';
+
+export const SPEECH_TRACKS = ['a1', 'a4'];
+const NBSP = ' ';
+const clean = value => String(value ?? '').trim();
+const end = clip => clip.start + clip.frames;
+
+// What a chosen stretch asks to be dubbed: a sound clip of the original sound (or its separated voice), whole or the
+// part a stretch covers, in seconds from the clip's start. Null for anything else.
+export function voiceTarget(record, selection) {
+ if (!selection) return null;
+ let track, clip, from, to;
+ if (selection.kind === 'clip') {
+ // A sound clip is chosen together with the picture it is linked to; exactly one speech clip may be among them.
+ // Choosing a picture clip (which brings its linked sound along) is not choosing the voice.
+ if (record.tracks.some(row => row.kind !== 'audio' && row.clips.some(value => selection.ids.includes(value.id)))) return null;
+ const found = record.tracks.filter(row => SPEECH_TRACKS.includes(row.id)).flatMap(row => row.clips.filter(value => selection.ids.includes(value.id)).map(value => [row, value]));
+ if (found.length !== 1 || !found[0][1].source) return null;
+ [track, clip] = found[0];
+ from = 0; to = clip.frames;
+ } else if (selection.kind === 'range') {
+ if (selection.tracks?.length !== 1 || !SPEECH_TRACKS.includes(selection.tracks[0])) return null;
+ track = record.tracks.find(row => row.id === selection.tracks[0]);
+ clip = track?.clips.find(value => value.source && value.start <= selection.start && end(value) >= selection.start + selection.frames);
+ if (!clip) return null;
+ from = selection.start - clip.start; to = from + selection.frames;
+ } else return null;
+ return {clipId: clip.id, track: track.id, separated: track.id !== 'a1', start: from / FPS, end: to / FPS, clip};
+}
+
+// The lines whose text differs from the recognised one; spaces at the ends do not count.
+export const changedLines = (lines, texts) => lines.filter(line => clean(texts.get(line.id) ?? line.text) !== clean(line.text));
+
+// Seconds as the bar shows them: m:ss.
+export const clockText = seconds => { const whole = Math.max(0, Math.floor(seconds)); return `${Math.floor(whole / 60)}:${String(whole % 60).padStart(2, '0')}`; };
+
+// The button names what it does; the time is not shown: the figure the service has is the new sentence's length, not the wait.
+export function redubLabel(count, t) {
+ if (!count) return t('Redub', '重新配音');
+ return t(`Redub ${count} ${count === 1 ? 'line' : 'lines'}`, `重新配音 ${count} 句`);
+}
+
+// One line needs no number; the time only once the service has given one.
+export const dubbingText = (index, count, seconds, t) => {
+ const time = Number.isFinite(seconds) ? ` · ${t(`about ${Math.max(1, Math.ceil(seconds))}${NBSP}s`, `约 ${Math.max(1, Math.ceil(seconds))}${NBSP}秒`)}` : '';
+ return count > 1 ? t(`Dubbing · line ${index} of ${count}${time}`, `正在配音 · 第 ${index}/${count} 句${time}`)
+ : t(`Dubbing${time}`, `正在配音${time}`);
+};
+
+// "2.6 s too long. Original voice kept.": one decimal, never "0.0". Without a way to shorten it, the row says what to do.
+export const overrunText = (seconds, t, canShorten = true) => {
+ const value = Math.max(0.1, Math.round(seconds * 10) / 10).toFixed(1);
+ return canShorten ? t(`${value}${NBSP}s too long. Original voice kept.`, `超出 ${value}${NBSP}秒,已保留原声`)
+ : t(`${value}${NBSP}s too long. Original voice kept. Shorten this line.`, `超出 ${value}${NBSP}秒,已保留原声。请缩短这句。`);
+};
+
+export const failedText = (count, t) => count === 1
+ ? t('1 line could not be dubbed. Redub it.', '有 1 句无法配音,请重新配音。')
+ : t(`${count} lines could not be dubbed. Redub them.`, `有 ${count} 句无法配音,请重新配音。`);
+
+// What a result says about each line it dubbed: kept the original voice because it did not fit, failed, or placed.
+export function lineResults(result) {
+ const out = new Map();
+ // A line the run was stopped before is simply not dubbed: no mark.
+ for (const line of (result?.lines || []).filter(value => value.code !== 'cancelled')) out.set(line.id, line.code ? {state: 'failed', code: line.code}
+ : line.replaced === false ? {state: 'kept', overrun: line.overrun || 0}
+ : {state: 'placed', overrun: 0, short: line.warning === 'too_short'});
+ return out;
+}
diff --git a/web/editor_voice_store.js b/web/editor_voice_store.js
new file mode 100644
index 0000000..2a654f2
--- /dev/null
+++ b/web/editor_voice_store.js
@@ -0,0 +1,64 @@
+// The dubbing calls of the editor page, through the engine's agent routes like the sound separation: each one is a job
+// read until it ends. The job leaves the project alone (lay: false); the page puts the new sound in itself, so its own
+// saves never clash with the job and the change can be undone.
+import {agentGet, agentPost, retryAgentPost} from './agent_session.js';
+
+const ROOT = '/freevideo/agent/v1';
+const ACTIVE = ['waiting', 'queued', 'running', 'cancelling'];
+const wait = ms => new Promise(resolve => setTimeout(resolve, ms));
+const detail = progress => { try { return JSON.parse(progress?.detail || 'null') || {}; } catch { return {}; } };
+const failure = (value, fallback) => Object.assign(new Error(value?.code || value?.error?.code || fallback), {code: value?.code || value?.error?.code || fallback});
+
+export async function voiceAvailable(api) {
+ try {
+ const response = await agentGet(api, `${ROOT}/capabilities`);
+ return !!(response.ok && (await response.json())?.features?.['projects.voice']);
+ } catch { return false; }
+}
+
+export function createVoiceStore({api, state, queue}) {
+ let running = null;
+ const uuid = () => globalThis.crypto?.randomUUID?.() || `${Date.now()}-${Math.random()}`;
+ const path = name => `${ROOT}/projects/${encodeURIComponent(state.record.id)}/voice/${name}`;
+ const selection = found => ({clip_id: found.clipId, start: found.start, end: found.end});
+ async function saved() {
+ // A change made a moment ago is queued after its own event; let it be seen before asking.
+ for (let round = 0; round < 3; round++) { await new Promise(resolve => setTimeout(resolve, 0)); await queue.flush(); if (!queue.dirty) break; }
+ if (queue.dirty || !state.record.id) throw failure(null, 'unsaved');
+ }
+ async function job(name, body, progress = () => {}) {
+ await saved();
+ const project = state.record.id, tool = running = {job: null};
+ try {
+ const response = await retryAgentPost(api, path(name), {revision: state.record.revision, ...body, ref: uuid()});
+ const answer = await response?.json().catch(() => null);
+ let current = answer?.job || answer?.jobs?.[0];
+ if (!response?.ok || !current?.id) throw failure(answer, 'voice_failed');
+ tool.job = current.id;
+ while (ACTIVE.includes(current.status)) {
+ progress({phase: current.progress?.phase, status: current.status, ...detail(current.progress)});
+ const reply = await agentGet(api, `${ROOT}/jobs/${encodeURIComponent(tool.job)}?wait=25`);
+ if (!reply.ok) { await wait(1000); continue; }
+ current = await reply.json();
+ if (state.record.id !== project) throw failure(null, 'project_changed');
+ }
+ if (current.status === 'succeeded' || (current.status === 'cancelled' && current.result)) return current.result;
+ throw failure(current.error, current.status === 'cancelled' ? 'cancelled' : 'voice_failed');
+ } finally { if (running === tool) running = null; }
+ }
+ async function sync(name, body) {
+ const response = await agentPost(api, path(name), body);
+ const answer = await response.json().catch(() => null);
+ if (!response.ok) throw failure(answer, 'voice_failed');
+ return answer;
+ }
+ return {
+ lines: ({found, language}) => job('lines', {selection: selection(found), ...(language ? {language} : {})}),
+ estimate: ({lines}) => sync('estimate', {lines}),
+ replace: ({found, lines, recognition, progress}) => job('replace', {selection: selection(found), lines, recognition, lay: false}, info => {
+ if (info.phase === 'dubbing') progress({index: info.index, count: info.count, line: info.line, remaining_seconds: info.remaining_seconds});
+ }),
+ shorten: ({line, target_seconds}) => sync('shorten', {line, target_seconds}),
+ cancel: () => { if (running?.job) agentPost(api, `${ROOT}/jobs/${encodeURIComponent(running.job)}/cancel`, {}).catch(() => {}); },
+ };
+}
diff --git a/web/health.js b/web/health.js
index 2f7337a..3a515a4 100644
--- a/web/health.js
+++ b/web/health.js
@@ -14,7 +14,7 @@ const t = (en, zh) => cn ? zh : en;
const launchVisit = query.get('freevideo') === 'launch';
// Every script this folder ships; a test keeps the list in step with web/*.js.
export const MODULES = ['audio_original.js', 'chat_data.js', 'phone_model.js', 'phone.js', 'lan_access.js', 'qrcodegen.js', 'agent_activity.js', 'agent_confirm_settings.js', 'agent_connect.js', 'agent_session.js', 'autopilot.js', 'autopilot_chat.js', 'chat_attachments.js', 'autopilot_conversations.js', 'browser_extension.js', 'caret_rect.js', 'chain_support.js', 'chain_rules.js', 'timed_prompt.js', 'capability_view.js', 'cast_generate.js', 'cast_mention.js', 'cast_overlay.js', 'cast_source.js', 'compatibility.js', 'entry_showcase.js', 'error_panel.js', 'first_run_guide.js', 'fonts.js', 'freevideo.js', 'generation_progress.js',
- 'editor.js', 'editor_actions.js', 'editor_lifecycle.js', 'editor_adjust.js', 'editor_adjust_picture.js', 'editor_picture_fields.js', 'editor_picture_session.js', 'editor_preview_picture.js', 'editor_entry.js', 'editor_expand.js', 'editor_export_dialog.js', 'editor_first_frame.js', 'editor_media_references.js', 'editor_references.js', 'editor_genbar.js', 'editor_duration.js', 'editor_geometry.js', 'editor_library.js', 'editor_menu.js', 'editor_stems.js', 'editor_record.js', 'editor_record_model.js', 'editor_model.js', 'editor_frame.js', 'editor_player.js', 'editor_room.js', 'editor_select.js', 'editor_state.js', 'editor_store.js', 'editor_tracks.js', 'editor_generation.js', 'editor_transport.js', 'editor_transport_model.js', 'editor_audio.js', 'editor_audio_model.js', 'editor_audio_notes.js', 'editor_audio_replace.js', 'editor_audio_rules.js', 'editor_music.js', 'editor_music_model.js', 'editor_speed.js', 'editor_speed_model.js', 'editor_failure.js', 'editor_header_fit.js',
+ 'editor.js', 'editor_actions.js', 'editor_lifecycle.js', 'editor_adjust.js', 'editor_adjust_picture.js', 'editor_picture_fields.js', 'editor_picture_session.js', 'editor_preview_picture.js', 'editor_entry.js', 'editor_expand.js', 'editor_export_dialog.js', 'editor_first_frame.js', 'editor_media_references.js', 'editor_references.js', 'editor_genbar.js', 'editor_duration.js', 'editor_geometry.js', 'editor_library.js', 'editor_menu.js', 'editor_stems.js', 'editor_record.js', 'editor_record_model.js', 'editor_voice.js', 'editor_voice_model.js', 'editor_voice_store.js', 'editor_model.js', 'editor_frame.js', 'editor_player.js', 'editor_room.js', 'editor_select.js', 'editor_state.js', 'editor_store.js', 'editor_tracks.js', 'editor_generation.js', 'editor_transport.js', 'editor_transport_model.js', 'editor_audio.js', 'editor_audio_model.js', 'editor_audio_notes.js', 'editor_audio_replace.js', 'editor_audio_rules.js', 'editor_music.js', 'editor_music_model.js', 'editor_speed.js', 'editor_speed_model.js', 'editor_failure.js', 'editor_header_fit.js',
'editor_loader.js', 'editor_text_coverage.js', 'editor_text.js', 'editor_toolbar_more.js', 'editor_text_model.js', 'editor_text_layout.js', 'editor_notes.js', 'editor_icons.js', 'dialog_keys.js', 'done_notify.js', 'failure_issue.js', 'feedback.js', 'frame_sizes.js', 'health.js', 'image_commands.js', 'image_editor.js', 'image_editor_engine.js', 'image_generation.js', 'image_lora.js', 'launch_visit.js', 'launcher.js', 'length_plan.js', 'lora_triggers.js', 'lora_check.js', 'library.js', 'motion.js', 'native_errors.js', 'translate_guard.js', 'output_download.js', 'port_notice.js', 'preview_scene.js',
'progress_connection.js', 'prompt_draft.js', 'prompt_enhance.js', 'prompt_references.js', 'prompt_sections.js', 'prompt_terms.js', 'request_assets.js', 'repair_hint.js', 'report_issue.js', 'issue_button.js', 'studio_recent.js', 'task_groups.js',
'result_actions.js', 'result_reuse.js', 'sampling_effort.js', 'pass_mode.js','setup.js', 'share.js', 'share_layout.js', 'studio.js', 'studio_chain.js', 'studio_extend.js', 'studio_length.js', 'first_frame_row.js', 'replace_person.js', 'replace_state.js', 'optional_inputs.js', 'other_usage.js', 'studio_queue.js', 'studio_size.js',